Data Concepts

Weak Supervision

Nobody will hand-label a million rows. Write a dozen noisy rules instead, then model how often they are wrong and combine them into probabilistic labels.

Three noisy rules vote on a row and a label model weights each vote by the accuracy it estimated for that rule, which is why it beats a majority vote that treats a 60% rule as equal to an 80% one

Why Does This Exist?

You need labels for two million documents. Hand-labelling them is a year and a budget nobody will approve, and active learning helps with the ratio but you still need volume.

So write rules instead. "If it contains an unsubscribe link, call it marketing." "If the sender domain is in this list, call it internal." "If it mentions an invoice number pattern, call it billing." Each rule is wrong a meaningful fraction of the time, none covers everything, and they contradict each other.

That sounds worse than useless, and the insight that makes weak supervision work is that it isn't. If you can estimate how often each rule is wrong, you can combine their votes into a probabilistic label that's substantially better than any single rule. You've moved the human effort from labelling examples to writing rules — which scales, because one rule labels a million rows.

Think of It Like This

Three unreliable weather forecasters

Three people tell you whether it'll rain. One is right 80% of the time, one 70%, one 60%.

Take the majority and you do better than the 70% and the 60% forecaster. You do worse than just listening to the 80% one, because a majority vote gives the 60% forecaster the same weight as the best of them, and two weak voices outvote one good one.

Now suppose you kept records, so you know each person's hit rate. You'd weight them: the 80% forecaster's opinion counts more, and the 60% one only matters when the other two disagree.

The trick that makes this work at scale is that you don't need the records. How often three forecasters agree with each other already tells you roughly how accurate each one is — a forecaster who agrees with the others constantly is probably good, one who's frequently the odd voice probably isn't. No ground truth required.

How It Actually Works

Labelling functions

A labelling function takes an example and returns a label or abstains. That third option is essential: a rule should only fire where it has an opinion, and forcing a guess everywhere destroys its precision.

They come from anywhere. Keyword and regex matches. A heuristic over metadata. An existing knowledge base — matching entity names against a database, which is the older idea of distant supervision. A weaker off-the-shelf model. Another team's legacy rules. An LLM prompted to classify, which is now the most common source and is exactly a labelling function: cheap, broad coverage, and wrong in correlated ways.

Two properties describe each one. Coverage is the fraction of examples it fires on. Accuracy is how often it's right when it does. You want a set with complementary coverage, and you accept mediocre accuracy.

The label model, and why majority vote isn't enough

Given a matrix of votes — rows are examples, columns are labelling functions, cells hold a label or an abstain — a label model estimates each function's accuracy and produces a probabilistic label per row.

It does this without ground truth, from the agreement structure. If two functions agree far more often than chance, both are probably accurate. If one disagrees with everything, it's probably noise. That's a solvable estimation problem, and it's the technical core of the approach.

Majority vote is the degenerate case where every function is assumed equally accurate, and the arithmetic below shows what that costs.

Then you train your actual model on the probabilistic labels — a noise-aware loss, weighting each example by label confidence. The end model generalises past the rules, because it sees the raw features and the rules only saw their own triggers.

Correlated functions are the real limitation

The accuracy estimation assumes the functions err roughly independently. Two rules that are near-duplicates — two keyword lists overlapping heavily, or three prompts to the same LLM — violate that badly, and the label model reads their agreement as mutual confirmation and inflates both accuracies.

That's the failure to watch for, because it's invisible in the outputs: the label model reports high confidence and the labels are systematically wrong wherever those correlated rules fire. Modern label models let you declare known correlations, and you should.

Where it fits

Weak supervision is for volume. Hand labelling is for evaluation — you still need a real labelled test set, and it has to be human-labelled, because a test set built from the same rules just measures whether the rules agree with themselves. Budget a few hundred to a few thousand careful labels for that, and spend the rest of the effort on rules.

The loop that works: write rules, run the label model, look at where it's least confident and where functions conflict, write a rule for that region, repeat. Same shape as improving annotation guidelines, and the conflict sets are equally informative.

Worked example

Three labelling functions, all firing, accuracies 70%, 80% and 60%, erring independently.

A majority vote is right whenever at least two are right. Summing the cases gives 0.788. That's better than the 70% rule and the 60% rule, and worse than the 80% rule on its own — the weak rule dragged the ensemble below its best member.

Make the accuracies equal instead, all three at 70%, and the majority vote scores 0.784, comfortably above any individual rule. Same voting mechanism, opposite verdict, and the only thing that changed was how evenly the accuracy was spread.

Push it further. One good rule at 90% and two weak ones at 55%: majority vote gives 0.748, well below the 90% rule. Two coin-flippers reliably outvote one expert.

That's the whole argument for a label model. Majority voting is only defensible when your functions are of similar quality, and you don't know whether they are — which is what the label model is for.

Show Me the Code

1import numpy as np2from itertools import product3
4def majority_accuracy(accs: tuple[float, ...]) -> float:5    """Chance a majority vote is right, if the rules err independently."""6    total = 0.07    for outcome in product([True, False], repeat=len(accs)):8        p = np.prod([a if ok else 1 - a for a, ok in zip(accs, outcome)])9        if sum(outcome) * 2 > len(accs):10            total += float(p)11    return total12
13print(round(majority_accuracy((0.7, 0.7, 0.7)), 4))    # -> 0.784  beats every single rule14print(round(majority_accuracy((0.7, 0.8, 0.6)), 4))    # -> 0.788  loses to the 0.80 rule15print(round(majority_accuracy((0.9, 0.55, 0.55)), 4))  # -> 0.748  loses badly to the 0.90 rule

Three calls, three verdicts on the same mechanism. A label model exists to weight the votes so the third case stops happening.

Watch Out For

Labelling functions that are secretly the same rule

You write twelve functions and four of them are keyword lists with heavy overlap, or three are the same LLM prompt with different wording.

The label model estimates accuracy from agreement, and those four agree constantly — not because they're right, but because they're the same rule wearing four names. So it concludes all four are highly accurate and weights them heavily, and wherever they're jointly wrong the final label is confidently wrong.

Nothing in the output flags it. Confidence is high, coverage looks healthy, and the errors are concentrated exactly where the correlated block fires.

Two checks before trusting a label model. Compute the pairwise agreement matrix between your functions and look for pairs above roughly 0.9 — those are duplicates, so merge them or declare the correlation if your label model supports it. And check accuracy on your human-labelled test set sliced by which functions fired, because an aggregate number hides a region where a correlated block owns the label.

A test set built from the same rules

You've generated two million probabilistic labels. Holding out 10% for evaluation is the obvious move and it measures nothing.

Those held-out labels came from the same labelling functions, so a model that agrees with them is a model that learned your rules. Score 94% and all you know is that the model successfully imitated a dozen regexes — including everywhere they're wrong, which is the part you can't see.

The version that fools people has an extra step: the end model beats the label model on that holdout, which reads as generalisation. It can just as easily be the end model smoothing over the rules' noise in a way that agrees with the rules more often, and no held-out rule-generated label can distinguish those.

Hand-label a real test set before writing a single labelling function. A few hundred careful, adjudicated examples with a measured agreement rate is enough, and it's the only number in the project that means anything.

The Quick Version

Move the human effort from labelling examples to writing rules, because one rule labels a million rows.
A labelling function returns a label or abstains. Abstaining is essential — a rule forced to guess everywhere loses its precision.
Describe each function by coverage and accuracy. Aim for complementary coverage and accept mediocre accuracy.
A label model estimates each function's accuracy from how often they agree, with no ground truth, then emits probabilistic labels.
Majority vote assumes equal accuracy. Three rules at 70% vote to 0.784; at 70/80/60 they vote to 0.788, below the 80% rule; at 90/55/55 they vote to 0.748.
Train the end model on probabilistic labels with a noise-aware loss, and it generalises past the rules.
Correlated functions break the accuracy estimates and produce confidently wrong labels. Check pairwise agreement and declare known duplicates.
LLM prompts are labelling functions: broad coverage, correlated errors.
Your test set must be human-labelled. A rule-generated holdout only measures whether the rules agree with themselves.