trackio/pr_708/en/storage_schema.md

Storage Schema and Direct Queries

Trackio stores each project in its own SQLite database and can export that data to parquet for syncing, static Spaces, and direct analysis. This page documents the live SQLite schema, the derived parquet layout, and the new trackio query command for running read-only SQL.

Where Data Lives

• Local project databases live in TRACKIO_DIR, which defaults to ~/.cache/huggingface/trackio.
• Each project is stored as a separate SQLite file: {project}.db.
• Media and uploaded files live under TRACKIO_DIR/media/.
• Each trace is additionally written as a Hub-renderable agent session file under TRACKIO_DIR/traces/ (on a Space, the bucket's traces/ prefix). These are a rendering artifact for the Hub's agent trace viewer; the traces table stays authoritative. See Traces.
• When syncing to Hugging Face, Trackio exports parquet files from the SQLite database before upload.

Querying Data Directly

Use the CLI when you need a catch-all query that is not already covered by trackio list or trackio get:

trackio query project --project "my-project" --sql "SELECT run_name, MAX(step) AS last_step FROM metrics GROUP BY run_name ORDER BY last_step DESC"

Add --json for machine-readable output:

trackio query project --project "my-project" --sql "SELECT name FROM sqlite_master WHERE type = 'table' ORDER BY name" --json

The same command works against a remote Space:

trackio query project --project "my-project" --sql "SELECT COUNT(*) AS num_alerts FROM alerts" --space username/my-space --json

trackio query only accepts read-only SELECT, WITH, and safe schema PRAGMA queries, including reading PRAGMA user_version.

SQLite Schema

The live schema is defined in trackio/sqlite_storage.py by SQLiteStorage.init_db().

metrics

Column	Type	Notes
id	INTEGER	Primary key
timestamp	TEXT	ISO timestamp
run_name	TEXT	Run identifier
step	INTEGER	Training step
metrics	TEXT	JSON blob containing logged metric values
log_id	TEXT	Optional deduplication key added by additive migration
space_id	TEXT	Optional pending-sync marker added by additive migration

Indexes:

• idx_metrics_run_step on (run_name, step)
• idx_metrics_run_timestamp on (run_name, timestamp)
• idx_metrics_log_id unique partial index on log_id
• idx_metrics_pending partial index on space_id

configs

Column	Type	Notes
id	INTEGER	Primary key
run_name	TEXT	Run identifier
config	TEXT	JSON blob containing run config
created_at	TEXT	ISO timestamp

Indexes and constraints:

• UNIQUE(run_name)
• idx_configs_run_name on (run_name)

system_metrics

Column	Type	Notes
id	INTEGER	Primary key
timestamp	TEXT	ISO timestamp
run_name	TEXT	Run identifier
metrics	TEXT	JSON blob containing system metrics
log_id	TEXT	Optional deduplication key added by additive migration
space_id	TEXT	Optional pending-sync marker added by additive migration

Indexes:

• idx_system_metrics_run_timestamp on (run_name, timestamp)
• idx_system_metrics_log_id unique partial index on log_id
• idx_system_metrics_pending partial index on space_id

traces

Column	Type	Notes
id	TEXT	Primary key
run_id	TEXT	Stable run identifier
timestamp	TEXT	ISO timestamp when the trace was logged
run_name	TEXT	Run display name
step	INTEGER	Training or logging step
key	TEXT	Key used in the trackio.log() payload
trace_index	INTEGER	Position when traces were logged as a list
messages	TEXT	JSON OpenAI-style messages
metadata	TEXT	JSON trace metadata
search_text	TEXT	Flattened text used by trace search
log_id	TEXT	Optional source log deduplication key
space_id	TEXT	Optional pending-sync marker
spans	TEXT	JSON execution spans; defaults to []

Indexes:

• idx_traces_run_step on (run_id, step)
• idx_traces_run_timestamp on (run_id, timestamp)
• idx_traces_search on search_text

project_metadata

Column	Type	Notes
key	TEXT	Primary key
value	TEXT	Metadata value

pending_uploads

Column	Type	Notes
id	INTEGER	Primary key
space_id	TEXT	Destination Space
run_name	TEXT	Optional run name
step	INTEGER	Optional step
file_path	TEXT	Absolute local path
relative_path	TEXT	Relative media path
created_at	TEXT	ISO timestamp

alerts

Column	Type	Notes
id	INTEGER	Primary key
timestamp	TEXT	ISO timestamp
run_name	TEXT	Run identifier
title	TEXT	Alert title
text	TEXT	Optional alert body
level	TEXT	info, warn, or error
step	INTEGER	Optional training step
alert_id	TEXT	Optional deduplication key

Indexes:

• idx_alerts_run on (run_name)
• idx_alerts_timestamp on (timestamp)
• idx_alerts_alert_id unique partial index on alert_id

artifacts

Column	Type	Notes
id	INTEGER	Primary key
name	TEXT	Artifact name
type	TEXT	Free-form category such as model or dataset
description	TEXT	Optional description
created_at	TEXT	ISO timestamp

Indexes and constraints:

• UNIQUE(name)

artifact_versions

Column	Type	Notes
id	INTEGER	Primary key
artifact_id	INTEGER	References artifacts.id
version	INTEGER	Sequential version number starting at 0
manifest_digest	TEXT	SHA-256 hex digest of the canonical manifest JSON
manifest	TEXT	JSON blob listing manifest entries with path, digest, size, and optional ref
metadata	TEXT	Optional JSON metadata
size_bytes	INTEGER	Total size of the version's manifest entries (sum of size) in bytes
producer_run_id	TEXT	Optional identifier of the producing run
producer_run_name	TEXT	Optional name of the producing run
created_at	TEXT	ISO timestamp

Indexes and constraints:

• UNIQUE(artifact_id, version)
• UNIQUE(artifact_id, manifest_digest)

artifact_aliases

Column	Type	Notes
artifact_id	INTEGER	References artifacts.id
alias	TEXT	Alias name such as latest
artifact_version_id	INTEGER	References artifact_versions.id

Indexes and constraints:

• PRIMARY KEY (artifact_id, alias)

run_artifact_links

Column	Type	Notes
id	INTEGER	Primary key
run_id	TEXT	Optional run identifier
run_name	TEXT	Optional run name
artifact_version_id	INTEGER	References artifact_versions.id
direction	TEXT	input or output
created_at	TEXT	ISO timestamp

Indexes and constraints:

• CHECK(direction IN ('input', 'output'))
• idx_run_artifact_links_run on (run_id, run_name)
• idx_run_artifact_links_version on (artifact_version_id)
• idx_run_artifact_links_unique unique index on (run_id, artifact_version_id, direction)

Registry tables

Registry databases — projects named registry-<name>, created by Api.create_registry — contain the standard tables above plus the four tables below, defined in trackio/registry_storage.py by RegistryStorage.init_registry_db(). Collection version numbers come from collections.next_version, which only moves forward: a collection version is never reused, even after an unlink. Every registry mutation also appends a row to registry_events in the same transaction.

collections

Column	Type	Notes
id	INTEGER	Primary key
name	TEXT	Collection name, UNIQUE
type	TEXT	The single artifact type the collection accepts
description	TEXT	Optional description
next_version	INTEGER	Monotonic counter for collection version numbers
created_at	TEXT	ISO timestamp

collection_links

Column	Type	Notes
id	INTEGER	Primary key
collection_id	INTEGER	References collections(id)
collection_version	INTEGER	Version number within the collection (v0, v1, ...)
source_project	TEXT	Project the linked artifact version was logged in
source_artifact	TEXT	Name of the linked artifact
source_version	INTEGER	Version of the linked artifact
source_space_id	TEXT	Space holding the source version, NULL when it is local
source_bucket_id	TEXT	Bucket holding the source version, NULL when it is local
created_at	TEXT	ISO timestamp

Constraints:

• UNIQUE(collection_id, source_project, source_artifact, source_version) — a source version is linked at most once per collection
• UNIQUE(collection_id, collection_version)

collection_aliases

Column	Type	Notes
collection_id	INTEGER	References collections(id)
alias	TEXT	Alias name
link_id	INTEGER	References collection_links(id)

Constraints:

• PRIMARY KEY (collection_id, alias) — one version per alias per collection

registry_events

Column	Type	Notes
id	INTEGER	Primary key
event_uid	TEXT	<compact-utc>-<writer>-<seq>, unique; totally orders events across writers and makes replay idempotent
ts	TEXT	ISO timestamp
kind	TEXT	create, link, promote, update, or unlink
payload	TEXT	JSON blob describing the mutation

The log is the authoritative form of a registry: the tables above are a fold of it. A bucket-backed registry stores the same events as one object each (see the Registry guide) and materializes these tables locally.

How Metric Payloads Are Stored

• User metrics are stored as JSON text in metrics.metrics.
• System metrics are stored as JSON text in system_metrics.metrics.
• Run configuration is stored as JSON text in configs.config.
• Trackio's higher-level APIs deserialize these JSON blobs before returning them.

Common direct-query pattern:

trackio query project --project "my-project" --sql "SELECT timestamp, step, metrics FROM metrics WHERE run_name = 'run-1' ORDER BY step DESC LIMIT 5"

Schema introspection example:

trackio query project --project "my-project" --sql "PRAGMA table_info(metrics)"

Parquet Layout

Trackio derives parquet files from SQLite by flattening JSON columns into regular columns.

Local parquet exports

SQLiteStorage.export_to_parquet() writes:

• {project}.parquet from metrics
• {project}_system.parquet from system_metrics
• {project}_configs.parquet from configs
• {project}_traces.parquet from traces
• {project}_{table}.parquet for each artifact table (artifacts, artifact_versions, artifact_aliases, run_artifact_links)

The flattened parquet files keep the structural columns such as timestamp, run_name, and step, then add one column per key found in the JSON payload.

Static Space and dataset exports

SQLiteStorage.export_for_static_space() writes:

• metrics.parquet
• aux/system_metrics.parquet
• aux/configs.parquet
• aux/traces.parquet
• aux/artifacts.parquet, aux/artifact_versions.parquet, aux/artifact_aliases.parquet, and aux/run_artifact_links.parquet
• runs.json
• settings.json

Static deployment copies media files to media/ and artifact blobs to artifacts/blobs/sha256/{prefix}/{digest} alongside these exports, so the browser-only dashboard can serve artifact downloads directly from the dataset or bucket.

This is the layout used for static Spaces and exported datasets.

Useful Query Examples

List tables:

trackio query project --project "my-project" --sql "SELECT name FROM sqlite_master WHERE type = 'table' ORDER BY name"

Find the latest step for each run:

trackio query project --project "my-project" --sql "SELECT run_name, MAX(step) AS last_step FROM metrics GROUP BY run_name ORDER BY last_step DESC"

Inspect recent alerts:

trackio query project --project "my-project" --sql "SELECT timestamp, run_name, level, title, step FROM alerts ORDER BY timestamp DESC LIMIT 20"

Inspect stored configs:

trackio query project --project "my-project" --sql "SELECT run_name, created_at, config FROM configs ORDER BY created_at DESC"

Schema Versions and Migrations

Each project database records its schema version in PRAGMA user_version:

trackio query project --project "my-project" --sql "PRAGMA user_version"

SQLiteStorage.init_db() brings a database up to SCHEMA_VERSION (defined in trackio/sqlite_storage.py) whenever it is opened for writing, and a Trackio server migrates every project database when it starts. Databases written before versioning existed report version 0 and are brought to version 1 by the additive CREATE TABLE IF NOT EXISTS and ALTER TABLE ... ADD COLUMN statements in init_db(). Version 2 gives databases written before Trackio 0.24 a run_id on every run-scoped table: those versions identified a run only by its name, so each run name becomes one run with a new id.

To change the schema or rewrite existing data in a way those statements cannot express, add a step to _SCHEMA_MIGRATIONS and increment SCHEMA_VERSION:

def _split_loss_column(cursor):
    cursor.execute("ALTER TABLE metrics ADD COLUMN loss REAL")
    cursor.execute("UPDATE metrics SET loss = json_extract(metrics, '$.loss')")

SCHEMA_VERSION = 3
_SCHEMA_MIGRATIONS = {2: _migrate_add_run_ids, 3: _split_loss_column}

_SCHEMA_MIGRATIONS[n] upgrades a database from version n - 1 to n. Each step runs in its own savepoint together with its version bump, so a failed step leaves the database at the last version that completed. A database at a newer version than the running Trackio knows is left untouched.

trackio bump relies on this: after restarting a Space on the local Trackio version, it checks that every project database reports the local SCHEMA_VERSION. The e2e test in tests/e2e-spaces/test_space_bump.py runs every migration against data written by Trackio 0.21.0 and 0.37.1.

Stability Notes

Trackio is still in beta. The schema is documented here so humans and agents can query it directly, but future releases may evolve the schema and require migrations or regenerated local databases.