Skip to content
AI360Xpert
Core ML

Model Extraction

By repeatedly querying a commercial API and recording its answers, an attacker can train a cheap, local 'knock-off' model that perfectly mimics the expensive proprietary model.

An attacker sends thousands of random inputs to a proprietary API, collects the predictions, and uses them as a training dataset to build a local clone model.
An attacker sends thousands of random inputs to a proprietary API, collects the predictions, and uses them as a training dataset to build a local clone model.

Why Does This Exist?

Training a state-of-the-art machine learning model costs millions of dollars in compute, data licensing, and engineer salaries. To recoup that cost, companies hide the model behind an API and charge users per query (e.g., $0.01 per prediction).

However, because the API returns high-quality predictions, those predictions can be used as perfectly labeled training data. A competitor can send thousands of queries to the API, record the answers, and use that dataset to train their own local model. This is called Model Extraction (or Model Stealing). The attacker effectively steals the intellectual property of the expensive model for pennies, creating a knock-off that performs almost as well as the original without paying the training costs.

Think of It Like This

Reverse-engineering a master chef's secret recipe

Imagine a world-famous chef who sells a soup with a secret recipe for $50 a bowl.

A rival restaurant owner doesn't know the recipe. Instead, they buy 100 bowls of the soup, take them back to their lab, and analyze the chemical composition of each bowl. Using that data, they figure out exactly what ingredients the chef is using and in what ratios.

They then start selling the exact same soup for $5. The rival didn't steal the literal recipe card from the chef's safe; they extracted the recipe just by observing the final product.

How It Actually Works

The Teacher-Student Dynamic

Model extraction is essentially Knowledge Distillation executed by a hostile third party.

  1. The Target (Teacher): The proprietary, expensive model behind an API.
  2. The Attacker (Student): A local, untrained model.
  3. The Dataset: The attacker generates a massive set of synthetic, random, or unlabeled inputs. They send these inputs to the Teacher API. The API returns the probabilities or classifications. The attacker pairs their inputs with the API's outputs to create a labeled dataset.

The attacker then trains the Student model on this dataset. Because the Teacher model's predictions contain rich, generalized signals about the underlying task, the Student model learns the task rapidly.

High-Fidelity Extraction (Logits vs. Hard Labels)

Model extraction is incredibly efficient if the target API returns probabilities (logits or confidence scores) rather than just a hard label.

If you ask an image classifier what a picture of a dog is, and it returns ["Dog": 80%, "Cat": 19%, "Car": 1%], the attacker learns a massive amount of information about the model's decision boundaries (e.g., this particular dog looks slightly like a cat). If the API only returns the hard label "Dog", extraction is still possible, but it requires significantly more queries to reverse-engineer the boundaries.

Show Me the Code

# A conceptual script for stealing a model via an APIdef extract_model(target_api, student_model, unlabeled_data):    stolen_dataset = []        # 1. Query the target API to generate labels    for data_point in unlabeled_data:        # The attacker pays a fraction of a cent for this query        target_prediction = target_api.query(data_point)                # Save the input and the stolen label        stolen_dataset.append((data_point, target_prediction))            # 2. Train the knock-off model on the stolen dataset    train_local_model(student_model, stolen_dataset)        return student_model
# The student_model now mimics the target_api, bypassing future API fees.

Watch Out For

API Terms of Service are not a technical defense

Nearly all commercial ML APIs state: "You may not use the output of this API to train a competing AI model." However, this is a legal defense, not a technical one. Proving that a competitor trained their model on your API outputs is incredibly difficult, which is why extraction attacks remain a major threat to AI startups.

Watermarking limitations

Some companies try to defend against extraction by "watermarking" their model—intentionally returning wrong answers for specific, bizarre inputs. If a competitor's model also gets those bizarre inputs wrong in the exact same way, it proves they stole the data. However, attackers can often bypass watermarks by smoothing the data or using ensemble techniques.

The Quick Version

  • Model extraction (or stealing) occurs when an attacker uses a proprietary API's outputs as training data to build a local clone.
  • It is a form of hostile Knowledge Distillation that allows attackers to steal intellectual property for a fraction of the cost of training a model from scratch.
  • APIs that return rich probability scores (logits) are much easier to extract than APIs that return simple hard labels.
  • Defending against extraction is notoriously difficult; rate limiting and limiting the detail of API responses are the primary technical mitigations.
  • Knowledge Distillation explains the legitimate version of this process, where a company trains a small model to mimic their own large model.
  • Membership Inference covers another privacy attack where an attacker tries to determine if a specific piece of data was used in the training set.
  • Model Inversion is an attack where the adversary tries to reconstruct the original training data directly from the model's weights.

Related concepts