MatrAIx Persona 1M

999,847 personas, each described by 1,290 categorical attributes. 599,847 are derived from real records, 400,000 are synthetic. 10 Zstandard Parquet shards, 4.17 GB.

Read it with pyarrow, not datasets

Attributes are packed: one persona's 1,290 attributes are 645 bytes of 4-bit codes, low nibble first. datasets cannot open these files at all. Use pyarrow and decode against persona_codes.schema.json.

import json, pyarrow.parquet as pq

schema = json.load(open("persona_codes.schema.json"))["columns"]   # 1,290 entries
table  = pq.read_table("data/persona-1m-0000.parquet")             # 100,000 personas

def decode(attributes, null_bitmap):
    """One row -> {field id: value}. Missing attributes are omitted."""
    out = {}
    for i, col in enumerate(schema):
        if null_bitmap is not None and (null_bitmap[i // 8] >> (i % 8)) & 1:
            continue                                   # bit set = missing
        code = (attributes[i // 2] & 0x0F) if i % 2 == 0 else (attributes[i // 2] >> 4)
        if code < len(col["values"]):
            out[col["id"]] = col["values"][code]
    return out

person = decode(table["attributes"][1].as_py(), table["null_bitmap"][1].as_py())
person.get("age_bracket"), person.get("region")

Three things to get right:

• Use .get(), not [...]. Rows are sparse: 656 of 1,290 attributes are populated on average, and age_bracket for instance appears in about a fifth of rows. Missing means the source did not support it; nothing is imputed.
• In null_bitmap, a set bit means missing, LSB first. A null bitmap means nothing is missing in that row.
• attribute_overrides beats the decoded code. It holds exact values that fall outside the current codebook.

To filter without scanning 4 GB, indexes/postings.sqlite maps each value to the global row ids that carry it.

Columns

Files

What the viewer shows

The viewer cannot read the packed shards, so it is pointed at sample/ instead: 999 personas as rows, 990 attribute ids as columns, each cell the decoded value. All seven sources appear. Columns run densest first, and rows are ordered by how many attributes are populated, so the table opens full and thins out further down.

How much a persona carries depends on where it came from. Synthetic personas are complete by construction; a persona extracted from one Amazon review supports around 16 attributes. A blank cell is an attribute the source did not support, never an imputed one.

300 of the 1,290 fields are left out: the Dataset Viewer refuses more than 1,000 columns. The release carries all of them.

The release is 999,847 personas, not the 999 rows shown above. The row count on this page, and anything load_dataset returns, describes that sample.

Composition

Four dimensions are calibrated against 2024 global population margins: age_bracket and region from UN WPP 2024, gender_identity and urbanicity from UN and World Bank totals with a schema prior over the remaining categories. Build is deterministic for seed 20260720. audit.json reports achieved versus target share per category; RESULTS.md summarises the build.

Limits

• Not a representative sample of any population. Calibration matches one-dimensional margins among rows where the field is known. It does not fix the joint distribution or remove source-selection bias.
• "Human-grounded" is not "verified". Wiki, Amazon, Stack Overflow and PRISM attributes come from model extraction and can carry extraction errors; GSS and survey mappings depend on crosswalk quality. Descriptions are model-generated.
• The 60/40 human-to-synthetic split is a design choice, not an estimate of any real ratio.
• Under-18 records were removed. 153 of the original 508 Real Human Survey records declared an age bracket under 18 and are gone; every count here reflects that, which is why the total is 999,847 rather than 1,000,000.

Source licenses and terms continue to apply to the underlying data.