File size: 2,575 Bytes
be9e22a e97ae70 be9e22a e97ae70 be9e22a | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 | ---
license: apache-2.0
base_model: EleutherAI/pythia-160m-deduped
library_name: onnx
pipeline_tag: text-generation
tags:
- transformers.js
- onnx
- interpretability
- attention
- tuned-lens
---
# Pythia-160M Observable
Browser-oriented ONNX export of [`EleutherAI/pythia-160m-deduped`](https://huggingface.co/EleutherAI/pythia-160m-deduped) for the interactive laboratory in the Engenharia Assistida por IA course.
The repository contains two runtime artifacts:
- `model_observable_q4.onnx`, which returns final next-token logits, 13 residual-stream states, the full attention matrix and the selected query token's attention update for each of the 12 transformer layers;
- `model_tuned_lens_q8.onnx`, which applies the layer-specific translators from the pretrained [`AlignmentResearch/tuned-lens`](https://huggingface.co/spaces/AlignmentResearch/tuned-lens) artifact and decodes an intermediate state into vocabulary logits.
## Observable model inputs
- `input_ids`: int64 tensor shaped `[batch, sequence]`;
- `attention_mask`: int64 tensor shaped `[batch, sequence]`;
- `query_index`: int64 tensor shaped `[batch]`.
The graph returns `next_token_logits`, `hidden_state_00` through `hidden_state_12`, `attention_01` through `attention_12`, and `attention_output_01` through `attention_output_12`. Each attention tensor has shape `[batch, heads, query sequence, key sequence]`.
## Tuned lens inputs
- `hidden_states`: float32 tensor shaped `[states, 768]`;
- `layer_index`: int64 tensor shaped `[states]`, with values from 1 through 12.
Layer 12 uses the model's final normalization and unembedding without a translator. Earlier layers use the pretrained translator for that residual-stream position.
## Quantization
Transformer matrix multiplications use weight-only Q4. The separate input and output embedding matrices use row-wise int8. The tuned lens translators and unembedding also use row-wise int8. Activations and public outputs remain float32.
The export manifest records the numerical checks performed against the PyTorch model and the original tuned lens. Quantization can change close-ranking tokens, so this artifact is intended for teaching and inspection, not evaluation or production inference.
## Limitations
- Pythia-160M is a small base model, not an instruction-following model.
- Attention weights describe values computed inside the model. They do not prove that a source token caused a prediction.
- Intermediate predictions are diagnostic readouts from a tuned lens, not text generated by stopping the original model early.
|