docs/data-loading-cache_en.md

# Data loading and caching

This document describes the current implementation of data loading and caching in the
HiveTrace Guardrail Leaderboard frontend.

The implementation has three goals:

- keep the private Hugging Face token on the SvelteKit server;
- avoid downloading and adapting unchanged bucket payloads;
- make repeated client-side navigation fast while providing a predictable data freshness window.

## Architecture

```text
Browser
  -> SvelteKit page or __data.json request
  -> Hugging Face Space Node process
  -> private Hugging Face bucket
       -> latest/manifest.json
       -> payload files referenced by the manifest
```

The browser never reads the private bucket directly. All bucket requests are made by the
SvelteKit server with the server-only `HF_TOKEN` environment variable.

## Data source

The bucket is configured with these runtime variables:

```text
HF_TOKEN=<private read token>
HF_BUCKET_ID=hivetrace/leaderboard_frontend_v2
HF_BUCKET_PREFIX=latest
HF_BUCKET_REQUEST_TIMEOUT_MS=60000
HF_BUCKET_REQUEST_RETRIES=2
HF_BUCKET_ENDPOINT=https://huggingface.co
```

`HF_BUCKET_ENDPOINT`, `HF_BUCKET_REQUEST_TIMEOUT_MS`, and `HF_BUCKET_REQUEST_RETRIES` are optional.
The values above are their defaults.

`HF_BUCKET_CACHE_TTL_MS` still controls the regular in-memory manifest cache and the direct
Tools snapshot helper. The main Ranking, Details, and Visualizations page-data flow explicitly
refreshes the manifest after the browser cache misses, so this variable does not add another
freshness delay to normal client-side page transitions.

## Manifest as the version pointer

`latest/manifest.json` is the single version pointer used by the frontend. Its important fields
are:

- `snapshot_id`: the identity of the published dataset;
- `files`: paths to all payload files;
- `hashes`: SHA-256 hashes of the payload files;
- `schema_version`: the bucket contract version;
- model, group, and dataset counts used for validation.

The frontend decides whether payload data changed by comparing `snapshot_id`. A new publish
must always use a new, unique `snapshot_id`.

Changing payload files or hashes without changing `snapshot_id` is not supported. In that case,
the frontend considers the snapshot unchanged and can continue using the old in-memory payload
until the Space process restarts.

## Routes and payloads

| Route          | Server loader               | Bucket data                                                                                                       |
| -------------- | --------------------------- | ----------------------------------------------------------------------------------------------------------------- |
| `/`            | `getHfBucketRankingState()` | catalog, leaderboard, and details matrix                                                                          |
| `/details`     | `getHfBucketDetailsState()` | catalog, leaderboard, and details matrix                                                                          |
| `/tools`       | `getHfBucketToolsState()`   | catalog, leaderboard, drilldown index, radar, scatter, heatmap, grouped bars, Pareto, performance, and robustness |
| `/methodology` | no bucket page loader       | no bucket payload                                                                                                 |

Ranking also uses the details matrix because the `partial` model status is derived from
`metrics_evaluated_samples < sample_count`.

The root `+layout.server.ts` reads the manifest separately to expose public bucket status such
as snapshot date and model count. That status uses the regular manifest cache. It is metadata
for the layout and does not control page-data freshness.

## Client-side navigation

SvelteKit intercepts internal menu links and requests route data through endpoints such as:

```text
/__data.json
/details/__data.json
/tools/__data.json
```

Localized variants are recognized as well, for example `/ru/details/__data.json`.

The response optimization in `src/hooks.server.ts` applies only to these three page-data
routes. It does not apply to Methodology, static assets, errors, or arbitrary API responses.

### Browser cache

Successful page-data responses use:

```http
Cache-Control: private, max-age=300
```

This means:

- each browser can reuse a route response for five minutes;
- the response is not a public CDN or shared-proxy cache entry;
- no `stale-while-revalidate` window is used;
- after five minutes, the next navigation must contact the SvelteKit server before rendering
  that route.

The cache is per response URL, including SvelteKit query parameters. Ranking, Details, and
Visualizations therefore have independent browser cache entries.

An already rendered page does not update itself. Fresh data is applied on a later navigation or
full page reload.

### Gzip compression

The same hook compresses a page-data response when all conditions are true:

- the response is successful and has a body;
- its declared size is at least 1,024 bytes;
- the client accepts `gzip`.

The compressed response contains:

```http
Content-Encoding: gzip
Vary: Accept-Encoding
```

`Content-Length` is removed because the compressed body is streamed. Clients without gzip
support receive the original body with the same five-minute private cache policy.

Approximate measured sizes for the current dataset are:

| Route data     | Uncompressed |   Gzip |
| -------------- | -----------: | -----: |
| Ranking        |       171 KB |  58 KB |
| Details        |       643 KB | 158 KB |
| Visualizations |       709 KB | 122 KB |

These sizes depend on the bucket contents and will change as models and datasets are added.

## Server request flow

When the browser does not have a fresh page-data response, the server follows this flow:

1. The page loader calls the corresponding Ranking, Details, or Tools state function.
2. The state function calls `fetchBucketRankingSnapshot({ refreshManifest: true })`.
3. `refreshManifest: true` bypasses the regular manifest TTL and reads the current small
   `manifest.json` from Hugging Face.
4. If the returned `snapshot_id` matches the in-memory ranking snapshot, catalog and leaderboard
   are reused without another payload download.
5. If `snapshot_id` changed, catalog and leaderboard are downloaded in parallel, their hashes
   and schemas are validated, and the ranking snapshot is replaced.
6. Route-specific data is reused or rebuilt for the same new `snapshot_id`.
7. SvelteKit serializes the adapted route data, and the server hook applies the private cache
   header and gzip compression.

The important distinction between the snapshot options is:

- `refreshManifest: true` always checks the manifest but reuses payloads when `snapshot_id` is
  unchanged;
- `forceRefresh: true` also bypasses payload reuse and rebuilds the ranking snapshot.

Normal page transitions use `refreshManifest`, not `forceRefresh`.

## In-memory caches

All server caches are module-level memory in the running Node process. They are private to a
single Space replica and are cleared when the container restarts, sleeps, or is redeployed.

### Manifest cache

`src/lib/server/hf-bucket/cache.ts` stores:

- the parsed manifest;
- the time it was fetched;
- its expiration time;
- one shared in-flight manifest request.

Regular callers use `HF_BUCKET_CACHE_TTL_MS`, with a five-minute default. Page-data refreshes use
the forced manifest path described above. Concurrent forced checks join the same in-flight
request when they overlap.

### Ranking snapshot

The ranking snapshot contains the manifest, catalog, and leaderboard. It is cached by
`snapshot_id`.

Concurrent downloads of the same snapshot share one promise. In-flight requests are also keyed
by `snapshot_id`, so a request for a newly published snapshot does not accidentally reuse a
download for an older snapshot.

### Details snapshot

The details snapshot adds `details_matrix` to the ranking snapshot and is cached by
`snapshot_id`.

This cache is shared by Ranking and Details. Ranking needs the matrix for `partial` detection, so
opening Details after Ranking does not download and validate `details_matrix` a second time.
Concurrent requests for the same details snapshot also share one promise.

### Adapted Ranking and Details datasets

The UI-ready Ranking and Details objects are cached by `snapshot_id`. When the manifest is
unchanged, the server returns the already adapted object. A new snapshot is parsed, validated,
and adapted once per Node process.

### Visualizations snapshot and adapted Tools dataset

For a new snapshot, the visualization files are downloaded in parallel. The raw visualization
snapshot and the adapted Tools dataset are cached separately by `snapshot_id`.

The direct `getHfBucketToolsSnapshot()` helper retains its TTL-based fast path. The `/tools`
page uses the state path, which refreshes the manifest after a browser cache miss.

## Validation

Payload reuse is allowed only after the manifest version check. New payloads go through:

- JSON parsing;
- schema parsing;
- SHA-256 verification when a hash is present;
- manifest count checks;
- model, group, and dataset reference checks;
- matrix completeness and duplicate-pair checks;
- visualization reference checks.

A validation error is handled in the same way as another refresh failure.

## Data freshness guarantee

Assume a valid new snapshot is published at time `T` and the bucket remains reachable.

### Route response is already in the browser cache

The old route response can be used until its individual five-minute `max-age` expires. The first
navigation to that route after expiration reads the fresh manifest. If `snapshot_id` changed,
the same navigation waits for and receives the new payload.

Therefore, the expected guarantee is:

> No later than the first navigation after the route's five-minute browser cache expires, the
> user receives the new snapshot.

There is no additional stale response window.

### Route response is not in the browser cache

The navigation immediately contacts the server and checks the manifest. If the server already
has the new payload, it reuses it. Otherwise, the navigation waits while the new snapshot is
downloaded and adapted.

### Full page reload

The five-minute policy targets SvelteKit `__data.json` navigation responses. A normal full HTML
request is not covered by this page-data cache and runs the server page loader again.

### Exceptions

The five-minute expectation does not apply when:

- Hugging Face or the network is unavailable;
- the new payload fails parsing, hash checks, or validation;
- the publisher reused the old `snapshot_id`;
- the manifest was published before all referenced files became readable.

## Failure behavior

The cache is also a resilience mechanism:

- if a refresh fails and a usable route dataset exists, the server returns cached data;
- if a new manifest is readable but a new payload is invalid or unavailable, the route state is
  returned as `stale` and the UI can show a warning;
- if no usable cache exists, the route returns an `unavailable` state with an empty fallback
  dataset;
- simultaneous requests share in-flight work to avoid duplicate bucket downloads.

One implementation detail is worth noting: when a forced manifest read fails but an older
manifest is cached, the manifest layer can return that stale manifest. The existing route data
can then continue to be served. The current route state does not always propagate this specific
manifest fallback as a visible `stale` warning.

## Publishing a new snapshot

The producer must publish atomically from the frontend's point of view:

1. Generate all payload files.
2. Use immutable or snapshot-specific file paths where practical.
3. Calculate and write the SHA-256 hashes.
4. Upload every payload file and verify it is readable.
5. Create a manifest with a new unique `snapshot_id`, correct paths, hashes, counts, and schema
   version.
6. Upload `latest/manifest.json` last.

Publishing the manifest last prevents the frontend from seeing a new version that references
files which are not available yet.

## Cold starts and replicas

After a Space restart or cold start, no in-memory snapshot exists. The first request must read
the manifest and all payloads required by that route. Later requests reuse the prepared data.

If the Space runs multiple replicas, each replica has its own memory cache and warms
independently. The browser cache remains local to each user.

## Operational checks

To inspect the page-data response headers, use an authenticated request to the Space:

```bash
curl -sS \
  -H "Authorization: Bearer $HF_TOKEN" \
  -H "Accept-Encoding: gzip" \
  -D - \
  -o /dev/null \
  "https://<space-domain>/details/__data.json"
```

Expected headers include:

```http
Cache-Control: private, max-age=300
Content-Encoding: gzip
Vary: Accept-Encoding
```

`curl` does not reproduce the browser's navigation cache unless an explicit curl cache is used.
Repeated curl requests therefore reach the server and can trigger repeated manifest checks.

When verifying a publish, check all of the following:

- the manifest has a new `snapshot_id`;
- every referenced file exists;
- hashes match the published contents;
- the Space logs contain no bucket validation errors;
- the first route navigation after browser cache expiration shows the new snapshot.

## Implementation map

| Responsibility                               | File                                                                                                   |
| -------------------------------------------- | ------------------------------------------------------------------------------------------------------ |
| HTTP bucket client and server-only token     | `src/lib/server/hf-bucket/client.ts`                                                                   |
| Regular manifest cache and in-flight request | `src/lib/server/hf-bucket/cache.ts`                                                                    |
| Ranking snapshot download and validation     | `src/lib/server/hf-bucket/ranking-snapshot.ts`                                                         |
| Shared details snapshot                      | `src/lib/server/hf-bucket/details-snapshot.ts`                                                         |
| Visualization snapshot download              | `src/lib/server/hf-bucket/tools-snapshot.ts`                                                           |
| Adapted Ranking cache                        | `src/lib/server/hf-bucket/ranking-cache.ts`                                                            |
| Adapted Details cache                        | `src/lib/server/hf-bucket/details-cache.ts`                                                            |
| Raw and adapted Tools caches                 | `src/lib/server/hf-bucket/tools-cache.ts`                                                              |
| Browser cache headers and gzip               | `src/hooks.server.ts`                                                                                  |
| Root public bucket status                    | `src/routes/+layout.server.ts`                                                                         |
| Route orchestration                          | `src/routes/+page.server.ts`, `src/routes/details/+page.server.ts`, `src/routes/tools/+page.server.ts` |