Data Concepts

Structured, Semi-Structured and Unstructured Data

Three kinds of data, split by one question: does the shape of a record exist before the data, travel inside each record, or not exist in writing at all?

The same complaint as a table row, a JSON event and a paragraph of prose, with the schema drawn where it lives in each: declared before the data, carried inside the record, or never declared at all so an encoder has to invent it

Why Does This Exist?

Two folders on one shared drive. One holds a spreadsheet of 40,000 orders, 22 columns, every cell the type it's meant to be. The other holds 40,000 support emails.

Both get called "the data". A question about the first takes a query and thirty seconds. The second takes a model and a fortnight.

The gap isn't size and it isn't mess. One question splits them: who supplies the schema, and when? A schema is the list of fields, their types, and what's allowed in them.

Three answers, and they're the whole taxonomy. Structured means the schema comes first and the database enforces it. Semi-structured, so JSON and event logs, means the schema travels inside each record and records may disagree. Unstructured, so text and images and audio, means nothing declares one, so a learned encoder invents it. That last case is why deep learning took these modalities: nobody could hand-write the fields for a photograph.

And "unstructured" is a bad name. A sentence has grammar, a photograph has geometry, a recording has harmonic structure. The structure is there, just not written anywhere a program can read.

Think of It Like This

A form, a note, and a conversation

Three ways to tell a hospital receptionist what's wrong with you.

You fill in a form. Name, date of birth, symptom from a dropdown of forty. Someone designed those boxes last year, before you walked in. Any receptionist can file and count it.

You hand over a note you wrote yourself, with headings you invented. "Pain, since Tuesday. Meds, two of them." The next patient's note has different headings. Filing still works, but every note has to be read first, and one new heading breaks whatever reads them.

Or you talk for three minutes. Nothing is labelled. Everything is in there, including the part that mattered most, and someone trained has to decide afterwards what the fields were.

Same complaint, three times. All that moved is where the boxes came from.

How It Actually Works

Three tiers, and what each one lets you do.

Structured, where the schema arrives first

Rows and columns, types declared up front, constraints checked on write. NOT NULL rejects the blank, a foreign key rejects the orphan, a type check rejects the string in an integer column. All of it on the way in, so bad records never land.

What that buys: index any column, join on any key, validate before insert, aggregate in one query. And feature engineering starts almost done, because the feature matrix is nearly the table already.

Semi-structured, where the schema rides along

JSON, XML, event logs. Each record carries its own keys, so the reader discovers the shape instead of being handed it. Two records in one file can hold different fields and neither is invalid.

You can still index and query, through path expressions rather than column names, and join once you've extracted the key you want. Validating means writing the contract yourself, because nothing upstream is.

Which is why this tier breaks pipelines rather than models. The model never sees the JSON. The flattening step does, and that's what fails, usually the morning a producer adds a field.

Unstructured, where no schema exists

Text, images, audio, video. Nothing declares fields, so you can't index on content, join on it, or aggregate it until something turns it into numbers.

That something is a learned encoder, and what it emits is an embedding: a fixed-length vector whose positions carry meaning the encoder worked out rather than meaning a person assigned. Hand-written rules for pixels never worked. Learning the schema beats declaring it.

Worked example

One customer complaint, three shapes:

1row     complaint_id=8814, category="billing", channel="email", days_open=4, csat=22
3event   {"id": 8814, "channel": "email",4         "steps": [{"t": "opened"}, {"t": "reply"}, {"t": "reply"}]}5
6prose   "Charged twice for October. Took four days and two calls, and the second7         agent contradicted the first."

The row gives you a group-by and a fitted model in one line, capped at the forty categories somebody picked in 2023. The event adds the sequence, how many replies and how long between them, and every field you pull from it needs extracting then defending. The prose holds the reason, which neither of the others contains, and holds nothing usable until an encoder turns it into vectors. Then it surfaces a double-charging complaint nobody gave a category to.

Show Me the Code

Four events, four shapes. Watch what flattening does to the column list.

1import numpy as np2
3events: list[dict[str, float]] = [4    {"id": 1.0, "amount": 42.0},5    {"id": 2.0, "callback_minutes": 6.0},6    {"id": 3.0, "amount": 19.0, "refund_ok": 1.0},7    {"id": 4.0},8]9
10
11def flatten(records: list[dict[str, float]]) -> tuple[list[str], np.ndarray]:12    """Every key any record used becomes a column. Absent means blank."""13    columns = sorted({key for record in records for key in record})14    present = [[key in record for record in records] for key in columns]15    return columns, np.array(present, dtype=float).mean(axis=1)16
17
18columns, fill = flatten(events)19print(columns)              # -> ['amount', 'callback_minutes', 'id', 'refund_ok']20print(fill.tolist())        # -> [0.5, 0.25, 1.0, 0.25]

Four records, four columns, one of them full. Scale to a real event stream and fill rates run to three decimals.

Watch Out For

Flattening nested JSON because the tooling lets you

pd.json_normalize over a week of events, and it works. That's the trap: it works on the sample.

Every key any record ever used becomes a column, so a payload with optional blocks turns 30 real attributes into 3,000 columns that are 99% blank. A dense frame stores every one of those blanks, and distance metrics drown, since two records sharing nothing but id still look alike across 2,998 matching nulls. Then a producer ships a new field, your column list changes shape, and the model answers on a vector it never trained on.

Stop treating the payload as a table. Pick the fields you want, name them in an explicit extraction step, leave the rest in the raw blob. If the sparsity is real, use a sparse matrix, and pin the expected shape with a data contract so the new field fails a check instead of shifting your columns.

Treating a free-text or ID column as categorical

user_id, session_token, error_message, product_sku. Strings that repeat, so a category encoder accepts them without complaint.

Then the cardinality shows up. One-hot on 200,000 user IDs gives 200,000 columns, each firing on a handful of rows, so the model memorises users instead of learning behaviour. Worse, the encoder's vocabulary is your training set: every production ID absent from training maps to unknown, right where you needed it most.

error_message fails the same way for a different reason: 40,000 distinct values because half contain a timestamp, so treating each as a level discards the 12,000 that say the same thing.

Sort the column by what it is, not by its dtype. A genuine low-cardinality category gets encoded. A high-cardinality identifier gets hashed, dropped, or turned into an aggregate like "orders in the last 30 days". Free text goes to a text pipeline.

The Quick Version

One question splits the three tiers: who supplies the schema, and when.
Structured: schema before the data, enforced on write. Index, join and aggregate stay cheap.
Semi-structured: schema inside each record, different per record. You discover it on read and validate it.
Unstructured: nothing declares a schema, so a learned encoder invents one and hands you vectors.
"Unstructured" doesn't mean structureless. Grammar and geometry are real, just not written anywhere readable.
Semi-structured data breaks pipelines, not models. The flattening step meets the new field first.
Flattening a payload blind gives thousands of near-empty columns and a schema that moves when a producer ships.
An ID column is a repeating string, not a category. Check cardinality first.