---
license: apache-2.0
base_model: EleutherAI/pythia-160m-deduped
library_name: onnx
pipeline_tag: text-generation
tags:
  - transformers.js
  - onnx
  - interpretability
  - attention
  - tuned-lens
---

# Pythia-160M Observable

Browser-oriented ONNX export of [`EleutherAI/pythia-160m-deduped`](https://huggingface.co/EleutherAI/pythia-160m-deduped) for the interactive laboratory in the Engenharia Assistida por IA course.

The repository contains two runtime artifacts:

- `model_observable_q4.onnx`, which returns final next-token logits, 13 residual-stream states, the full attention matrix and the selected query token's attention update for each of the 12 transformer layers;
- `model_tuned_lens_q8.onnx`, which applies the layer-specific translators from the pretrained [`AlignmentResearch/tuned-lens`](https://huggingface.co/spaces/AlignmentResearch/tuned-lens) artifact and decodes an intermediate state into vocabulary logits.

## Observable model inputs

- `input_ids`: int64 tensor shaped `[batch, sequence]`;
- `attention_mask`: int64 tensor shaped `[batch, sequence]`;
- `query_index`: int64 tensor shaped `[batch]`.

The graph returns `next_token_logits`, `hidden_state_00` through `hidden_state_12`, `attention_01` through `attention_12`, and `attention_output_01` through `attention_output_12`. Each attention tensor has shape `[batch, heads, query sequence, key sequence]`.

## Tuned lens inputs

- `hidden_states`: float32 tensor shaped `[states, 768]`;
- `layer_index`: int64 tensor shaped `[states]`, with values from 1 through 12.

Layer 12 uses the model's final normalization and unembedding without a translator. Earlier layers use the pretrained translator for that residual-stream position.

## Quantization

Transformer matrix multiplications use weight-only Q4. The separate input and output embedding matrices use row-wise int8. The tuned lens translators and unembedding also use row-wise int8. Activations and public outputs remain float32.

The export manifest records the numerical checks performed against the PyTorch model and the original tuned lens. Quantization can change close-ranking tokens, so this artifact is intended for teaching and inspection, not evaluation or production inference.

## Limitations

- Pythia-160M is a small base model, not an instruction-following model.
- Attention weights describe values computed inside the model. They do not prove that a source token caused a prediction.
- Intermediate predictions are diagnostic readouts from a tuned lens, not text generated by stopping the original model early.