Lewis-44M-Bert-V3

Lewis-44M-Bert-V3 is a compact (≈ 43.5M parameter) English encoder pre-trained from scratch with ELECTRA-style replaced-token detection on ~2.7B unique tokens (one epoch) of curated multi-domain text. It reuses the modernized architecture of Lewis-44M-Bert (RoPE, RMSNorm, GeGLU, QK-norm) and was trained entirely on a single consumer GPU (RTX 5050, 8 GB VRAM).

GLUE dev average: 80.5 (8 tasks, WNLI excluded), up from 75.2 for the MLM-trained v2 model at the same size (the fine-tuning recipe also changed; see Development history). On the official GLUE test server it scores 76.2 (9 tasks including WNLI; 77.6 over the same 8 tasks as the dev average, see GLUE test-server results). It has also been fine-tuned and evaluated on SQuAD v1.1 (83.8 F1 dev), CoNLL-2003 NER (87.8 test F1), topic and sentiment classification, and measured for inference speed.

Loading requires trust_remote_code=True (custom architecture).

The tokenizer adds the <s> … </s> wrapper (ids 2 and 3) for you, exactly as in pre-training. Only if you tokenize with your own code do you need to add them yourself: skipping them costs more than one GLUE point.

GLUE dev average	80.5
GLUE test-server score	76.2 (9 tasks incl. WNLI) / 77.6 (8 tasks, WNLI excluded)
SQuAD v1.1 dev (EM / F1)	75.2 / 83.8
CoNLL-2003 NER (entity F1, dev / test)	92.6 / 87.8
Parameters	≈ 43.5M (encoder only)
Objective	ELECTRA replaced-token detection (the discriminator is released)
Unique tokens / epochs	≈ 2.67B / 1
Hardware	1× RTX 5050 (8 GB)
License	Apache-2.0

---

Table of contents

1. Model details
2. Tech stack
3. Training data
4. Training procedure
5. Benchmarks
6. Inference speed
7. How to use
8. Why use it
9. Where to use it
10. Limitations and risks
11. Development history
12. Files, citation

---

Model details

Type	Bidirectional Transformer encoder (ELECTRA discriminator)
Parameters	≈ 43.5M encoder only (the MLM head of v1/v2, which brings the total to 43.8M, is not part of this model; embeddings were shared with the pre-training generator, which is discarded)
Layers	10
Hidden size	512
Attention heads	8 (head dim 64)
FFN	GeGLU, intermediate size 1152
Position encoding	Rotary (RoPE), theta = 10,000, computed in fp32
Normalization	Pre-norm RMSNorm (+ embedding norm, final norm)
Attention	Bidirectional SDPA with QK-norm, no biases
Vocabulary	30,000, cased byte-level BPE (no UNK)
Max sequence length	512
Init	N(0, 0.02); residual output projections scaled by 1/sqrt(2L)
Dropout	0.0 during pre-training

Special tokens: <pad>=0, <unk>=1, <s>=2 (BOS), </s>=3 (EOS), <mask>=4. Single sentence: <s> A </s>. Sentence pair: <s> A </s> B </s>. The tokenizer adds these automatically. The sequence-classification head pools position 0, which is why <s> matters.

What was released: only the ELECTRA discriminator, the encoder you fine-tune. The small generator used during pre-training is thrown away, as in the original ELECTRA recipe.

How ELECTRA pre-training works

Instead of masking tokens and asking the model to guess them (BERT/MLM), ELECTRA trains two networks:

1. A small generator (4-layer masked language model) fills randomly masked positions with plausible tokens.
2. The discriminator (the released model) looks at every token and decides: original or replaced?

The discriminator gets a learning signal from all tokens, not only the ~15% that were masked, so it learns more per training token.

---

Tech stack

Area	Tools
Framework	PyTorch 2.x, Hugging Face transformers (custom modeling + configuration classes), safetensors
Data	Pre-tokenized, packed uint16 binary files read through numpy.memmap; fixed 512-token chunks; deterministic, resumable sampler
Precision / speed	bfloat16 autocast, TF32 matmuls, fused AdamW, torch.compile on the backbone, gradient accumulation
Optimization	AdamW (β = 0.9, 0.98; ε = 1e-6), weight decay 0.01 (not on norms, biases, embeddings), gradient clipping 1.0, linear warmup then cosine decay
Checkpointing	Atomic saves, resumable training state, automatic pruning of old checkpoints
Evaluation	datasets (nyu-mll/glue, rajpurkar/squad, tomaarsen/conll2003), evaluate (SQuAD), seqeval (NER), scikit-learn and scipy for MCC / F1 / Pearson / Spearman, tqdm
Hardware	One NVIDIA RTX 5050 (8 GB VRAM), Windows. Pre-training throughput ≈ 34K tokens/s (≈ 22 hours for V3)

---

Training data

About 2.68B unique tokens, each seen once. Documents were tokenized with a tokenizer trained on the same mix, packed into 512-token chunks as <s> … </s>, and shuffled globally. 3,875 sequences (about 0.2%) are held out for validation, the same set used by v1 and v2.

Source	Tokens	Share	Purpose
FineWeb-Edu (score ≥ 3.5)	1,338.5M	50.0%	Dense, educational, analytical web text
Project Gutenberg	510.2M	19.1%	Long-form narrative, coreference, literary style
Wikipedia (English, 20231101)	490.1M	18.3%	Factual knowledge, named entities
News (cc_news)	200.1M	7.5%	Current-affairs prose
Cosmopedia (synthetic)	100.0M	3.7%	Procedural, textbook-style text
Q&A (ELI5-style)	38.5M	1.4%	Explanatory question answering
Total	≈ 2,677M	100%

The data was collected in two stages: ~1.0B tokens for v1 (FineWeb-Edu 50%, Wikipedia 25%, Gutenberg 15%, Cosmopedia 10%) and 1.68B new tokens for v2 (FineWeb-Edu, Gutenberg, Wikipedia, news, Q&A). Documents already used in the first stage were skipped, and exact duplicates were removed within the new data. V3 trains on both stages together, from scratch, for one epoch.

---

Training procedure

Stage	Objective	Data	Result
v1	MLM, 25% masking (80/10/10), from scratch	~1.0B unique tokens × 2 epochs (30,478 steps)	Val loss 1.987 (ppl 7.29); GLUE 75.0
v2	Continued MLM from v1 weights	1.68B new tokens × 1 epoch (25,569 steps)	Val loss 1.930 (ppl 6.89); GLUE 75.2
v3 (this model)	ELECTRA replaced-token detection, from random init	All ≈ 2.68B unique tokens × 1 epoch (40,809 steps)	GLUE 80.5

ELECTRA pre-training hyperparameters (V3)

Hyperparameter	Value
Sequence length	512
Batch	16 × 8 gradient accumulation = 128 sequences = 65,536 tokens per update
Optimizer steps	40,809 (≈ 2.67B tokens)
Peak LR → min LR	4e-4 → 4e-5 (cosine)
Warmup	6% (2,448 steps)
Generator mask rate	15%
Generator	4 layers, embeddings shared with the discriminator
Discriminator loss weight (λ)	50
Precision	bf16 autocast

Training diagnostics (step ~27,000): generator MLM loss ≈ 2.3, discriminator loss ≈ 0.19, ≈ 8.3% of tokens replaced, replaced-token precision ≈ 73%. Generator and discriminator stayed balanced throughout, with no divergence.

Compute: about 22 hours for V3 at ≈ 34K tokens/s on one RTX 5050 (8 GB). The whole project (v1 ≈ 21 h, v2 ≈ 10 h, V3 ≈ 22 h) took roughly 53 hours of single-GPU time.

Fine-tuning recipe used for the GLUE numbers

Task	LR	Layer-wise LR decay	Epochs	Batch	Seeds	Init
MNLI, QQP, QNLI	3e-4	0.8	3	32	1	pre-trained
RTE, MRPC, STS-B	1.5e-4	0.8	10	16	5 (median)	MNLI fine-tuned model
CoLA	5e-5	none	5	32	5 (median)	pre-trained
SST-2	5e-5	none	3	32	1	pre-trained

Other settings: AdamW, weight decay 0.01, warmup 6–10%, linear decay, max length 128, bf16, final-epoch score reported (no best-epoch picking), median seed reported. No dropout override was applied (the config default was used); hidden_dropout_prob=0.1 has not been tested.

---

Benchmarks

GLUE (dev set)

All GLUE numbers are dev-set results (8 tasks, WNLI excluded). They are not GLUE test-server scores.

Task	Metric	v2 (MLM)	V3 (ELECTRA)
CoLA	Matthews corr.	41.4	52.3
SST-2	Accuracy	89.9	88.6
MRPC	Acc / F1	78.4 / 84.9	85.3 / 89.4
QQP	Acc / F1	89.6 / 86.1	90.3 / 86.9
STS-B	Pearson / Spearman	83.5 / 83.2	87.7 / 87.5
MNLI	Acc (matched / mismatched)	79.0 / 79.2	79.9 / 79.8
QNLI	Accuracy	85.4	87.1
RTE	Accuracy	53.1	72.6
Average	8 tasks, WNLI excluded	75.22	80.51

v2 and V3 used different fine-tuning recipes (v2: basic single-run fine-tuning; V3: MNLI transfer, layer-wise LR decay, more epochs, multi-seed medians). The jump therefore combines the model and the recipe. See Development history for the part due to ELECTRA alone.

Under the ELECTRA paper's stricter convention (MCC for CoLA, Spearman for STS-B, accuracy elsewhere, MNLI-matched) the average is 80.45.

Seed variance (final-epoch score, 5 seeds):

Task	Mean	Std	Range
RTE	72.85	0.47	72.56 – 73.65
MRPC	87.43	1.08	85.78 – 88.71
STS-B	87.65	0.15	87.49 – 87.88
CoLA	51.74	1.63	48.84 – 52.69

MNLI, QQP, QNLI and SST-2 are single runs. Expect about ±0.2–0.3 on rerun (SST-2 can move more: its dev set has only 872 examples).

GLUE test-server results

Submitted to the official GLUE evaluation server (test labels are hidden). GLUE score: 76.2 (average over 9 tasks, MNLI counted as the mean of matched and mismatched).

Task	Metric	Test	Dev (table above)
CoLA	Matthews corr.	48.6	52.3
SST-2	Accuracy	88.0	88.6
MRPC	F1 / Accuracy	88.3 / 84.1	89.4 / 85.3
STS-B	Pearson / Spearman	85.7 / 85.4	87.7 / 87.5
QQP	F1 / Accuracy	69.7 / 88.4	86.9 / 90.3
MNLI-m / MNLI-mm	Accuracy	79.7 / 78.3	79.9 / 79.8
QNLI	Accuracy	87.1	87.1
RTE	Accuracy	67.3	72.6
WNLI	Accuracy	65.1	not trained (majority class)
AX (diagnostic)	Matthews corr.	30.1	n/a

• Averages: 76.2 over all 9 tasks. Over the same 8 tasks as the dev average (WNLI excluded, MNLI as the mean of matched and mismatched, and MRPC, STS-B and QQP as the mean of their two metrics) the test average is 77.6, against 80.5 on dev.
• WNLI was not trained. The submission predicts the majority class of the training set for every example, so it is not a model result.
• Largest dev-to-test gaps: RTE (−5.3, a very small test set), CoLA (−3.7, one of the two tasks whose settings were tuned on dev), and QQP F1 (69.7 against 86.9). QQP's test F1 is typically far below its dev F1 because the two sets are not balanced the same way, so QQP accuracy (88.4 against 90.3) is the more comparable number.
• How the predictions were produced: for RTE, MRPC, STS-B and CoLA (10, 10, 5 and 8 seeds) the test predictions come from the median-dev seed; SST-2, QNLI and QQP are single runs; MNLI and AX come from the MNLI model; RTE, MRPC and STS-B start from that MNLI model. The per-task learning rate, layer-wise decay and epochs were those in finetune_glue_v2.py, whose dev scores were within about one point of the dev table above.
• Single submission, no repeated leaderboard attempts.

Question answering: SQuAD v1.1 (dev set)

Metric	Score
Exact match	75.20
F1	83.83

Setup: a linear start/end span head on the encoder output, input <s> question </s> context </s>, max length 384 with a sliding window (stride 64) for long contexts, questions capped at 300 characters, AdamW (weight decay 0.01), LR 1e-4, 2 epochs, batch 32, 6% warmup then linear decay, bf16, standard best-span decoding (top 20 start/end candidates, answers up to 30 tokens). Single run, single seed, dev set only. The learning rate was not tuned.

Named-entity recognition: CoNLL-2003

Split	Entity-level F1
Validation	92.60
Test	87.83

Validation F1 by epoch: 90.85, 92.41, 92.51, 92.60. Setup: linear token-classification head, label on the first sub-token of each word, max length 256, AdamW (weight decay 0.01), LR 1e-4, 4 epochs, batch 32, 6% warmup then linear decay, bf16, entity-level F1 with seqeval. Single run, single seed. The final epoch was used (no best-epoch picking), and the test set was evaluated once. The test split of CoNLL-2003 is known to be harder than the validation split, which explains most of the gap between the two scores.

Other classification tasks (10k training examples)

Task	Classes	Max len	Test accuracy	Macro-F1
Emotion (dair-ai/emotion, 2,000 test)	6	64	90.40	85.35
AG News (fancyzhx/ag_news, 7,600 test)	4	128	90.30	90.30
IMDB (stanfordnlp/imdb, first 5,000 shuffled test reviews)	2	256	86.66	86.66

Setup: 10,000 randomly sampled training examples, the model's standard sequence-classification head, LR 5e-5, 3 epochs, batch 32, 6% warmup then linear decay, AdamW (weight decay 0.01), final-epoch scores, single seed (42). IMDB is truncated to 256 tokens, which limits accuracy on long reviews.

Low-data fine-tuning

Accuracy (mean ± std over 3 seeds) when fine-tuning on only n labeled examples. Same recipe for every size: LR 1e-4, batch 32, max length 128, 10 epochs for n ≤ 500, 5 epochs for n = 1,000, 3 epochs for n = 5,000.

n	SST-2 (872 dev examples)	AG News (2,000 test examples)
100	58.3 ± 1.4	66.3 ± 0.5
500	67.0 ± 1.1	81.2 ± 0.7
1,000	71.3 ± 0.2	84.2 ± 0.2
5,000	81.4 ± 0.6	89.9 ± 0.6

Reading this fairly: topic classification (AG News) is learned quickly from a few hundred examples. Sentiment (SST-2) needs more data, with the model near chance at 100 examples and in the low 80s at 5,000 (88.6 with the full training set). The learning rate and epoch counts were not tuned per size, so the smallest sizes may be under-trained.

What these results do not show

• The GLUE test-server result comes from a single submission, with the median-dev seed chosen for the multi-seed tasks, and WNLI was not trained.
• SQuAD, NER and the low-data runs are single-recipe (one learning rate) and, except where stated, single-seed.
• No retrieval, sentence-embedding or token-level tasks beyond CoNLL-2003 NER.
• CoLA and SST-2 settings were chosen after looking at earlier dev runs, so those two tasks are slightly tuned on dev.
• MNLI, QQP, QNLI and SST-2 are single-seed.
• Learning-rate sensitivity has not been measured; the numbers above come from the specific learning rates listed.

---

Inference speed

Encoder forward pass only (no tokenization), random token inputs, 30 timed iterations after warmup. GPU: NVIDIA RTX 5050 laptop GPU with bf16 autocast. CPU: laptop CPU, fp32, default PyTorch thread count. Peak memory is the GPU allocation during inference.

Device	Batch	Seq len	Latency per batch	Throughput	Peak GPU memory
GPU	1	128	16.7 ms	7.7K tok/s	246 MB
GPU	1	512	18.7 ms	27.4K tok/s	254 MB
GPU	32	128	39.0 ms	105K tok/s	327 MB
GPU	32	512	219 ms	75K tok/s	563 MB
CPU	1	128	84 ms	1.5K tok/s	n/a
CPU	1	512	262 ms	2.0K tok/s	n/a
CPU	32	128	1.04 s	3.9K tok/s	n/a
CPU	32	512	4.73 s	3.5K tok/s	n/a

• A single 128-token sentence takes about 17 ms on the GPU (limited by kernel-launch overhead at this size).
• Batched GPU throughput reaches about 105K tokens/s (roughly 800 sentences/s at 128 tokens).
• Inference peak memory stays under 600 MB even at batch 32 and 512 tokens.
• CPU timings vary with background load and clock speed: batch-1, 128-token latency ranged from about 50 ms to 85 ms across runs on the same machine. Treat CPU numbers as indicative.

---

How to use

Install

pip install torch transformers safetensors

Classification (the model has no MLM head)

Only the ELECTRA discriminator is released, so use it as an encoder. The classification head is new and randomly initialized: fine-tune before trusting its outputs.

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

repo = "vprojectx/Lewis-44M-Bert-V3"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(
    repo, trust_remote_code=True, num_labels=2
).eval()

# The tokenizer adds <s> ... </s> (and <s> A </s> B </s> for pairs) automatically.
batch = tok(["A great, funny movie.", "Terrible plot."],
            padding=True, truncation=True, max_length=128, return_tensors="pt")
with torch.no_grad():
    logits = model(**batch).logits

# sentence pairs (NLI, paraphrase, ...):  tok(premises, hypotheses, padding=True, ...)

If you use your own tokenization code, make sure every sequence starts with id 2 (<s>) and ends with id 3 (</s>), with pairs as <s> A </s> B </s>.

Sentence features (and a backbone for span / token tasks)

with torch.no_grad():
    hidden = model.Lewis_44M_Bert(
        batch["input_ids"], attention_mask=batch["attention_mask"], return_dict=True
    ).last_hidden_state
m = batch["attention_mask"].unsqueeze(-1).float()
embedding = (hidden * m).sum(1) / m.sum(1)      # mean pooling

hidden is the per-token output you can put a linear head on for question answering (start/end logits) or token classification (NER), which is how the SQuAD and CoNLL-2003 results above were produced.

This encoder was not trained with a sentence-similarity objective. For retrieval or semantic search, fine-tune it on pairs first.

Recommended fine-tuning settings

Task type	Settings that worked
Sentence classification (topic, sentiment, emotion)	LR 5e-5 to 1e-4, 3 epochs, batch 32, max length 64–256
Span QA (SQuAD)	LR 1e-4, 2 epochs, batch 32, max length 384, stride 64
Token classification (NER)	LR 1e-4, 4 epochs, batch 32, label first sub-token
GLUE sentence-pair tasks	See the GLUE recipe above (layer-wise LR decay 0.8, MNLI transfer for small tasks)

General tips from the GLUE work:

1. Layer-wise LR decay of 0.8, top LR about 1.5e-4–3e-4 (5e-5 with no decay worked best on CoLA and SST-2).
2. For small tasks (RTE, MRPC, STS-B), fine-tune on MNLI first and start from that checkpoint.
3. 10 epochs on small tasks, 3 on large ones, batch 16–32.
4. Several seeds on small datasets; report the median.
5. Always keep the <s> … </s> wrapper (the tokenizer does it for you).

Reproduce the GLUE evaluation

pip install torch transformers datasets scikit-learn scipy tqdm huggingface_hub
hf download vprojectx/Lewis-44M-Bert-V3 finetune_glue_final.py --local-dir .
python finetune_glue_final.py --out ./glue_final

The script downloads this model, trains all 8 tasks (MNLI first), writes results.json and seed_stats.json, and prints the GLUE average. It resumes if interrupted; the full run takes roughly 3–4 hours on a laptop-class GPU.

---

Why use it

• Small and cheap. 44M parameters. It fine-tunes on an 8 GB laptop GPU, runs on CPU, and needs under 600 MB of GPU memory for inference at batch 32 and 512 tokens.
• Covers the main task types. Sentence classification and inference (GLUE 80.5 dev, 77.6 test over 8 tasks), span extraction (SQuAD v1.1 F1 83.8) and token-level tagging (CoNLL-2003 NER F1 87.8 test).
• Fast. About 17 ms per sentence on a laptop GPU and about 105K tokens/s batched.
• Efficient pre-training. ≈ 2.7B tokens in one epoch, about 22 hours on one consumer GPU. Easy to extend, retrain or reproduce.
• Handles small, noisy datasets well such as RTE (72.6) and CoLA (52.3), where replaced-token detection helps.
• Modern architecture. RoPE, RMSNorm, GeGLU and QK-norm, which tend to improve stability and quality per parameter.
• Curated, open data. Educational-quality web text plus encyclopedic, literary, news, Q&A and synthetic sources; no distillation from a teacher model.
• Documented recipe. Training settings, fine-tuning settings, seed spread and the evaluation script are all published.

Where to use it

Good fits:

• Text classification (topic, intent, emotion, spam, toxicity after fine-tuning)
• Natural language inference and entailment checks
• Paraphrase and duplicate-question detection
• Semantic textual similarity (after fine-tuning)
• Extractive question answering and named-entity recognition
• Low-latency, on-device or CPU inference; budget-limited APIs
• A compact base for domain adaptation (legal, medical, support tickets) with a small labeled set
• Teaching, ablations and research baselines at small scale

Not a good fit:

• Text generation, chat, instruction following (encoder only)
• Documents longer than 512 tokens
• Non-English text
• Sentiment tasks with only a few hundred labeled examples (see the low-data results)
• Tasks where the last few points of accuracy matter: validate on your own data first
• High-stakes decisions without task-specific validation and human review

---

Limitations and risks

• Dev numbers are optimistic. The GLUE dev average (80.5) is higher than the GLUE test-server result (77.6 over 8 tasks, 76.2 over 9 including an untrained WNLI). SQuAD is dev-only. NER is reported on both validation and test.
• Light tuning on dev. The CoLA and SST-2 recipes were chosen after earlier dev runs; CoLA is 3.7 points lower on test than on dev.
• Single-seed results for MNLI, QQP, QNLI, SST-2, SQuAD, NER and the 10k-example classification runs. Differences of a few tenths of a point are within run-to-run noise.
• One learning rate per task for SQuAD, NER and low-data runs; sensitivity to the learning rate has not been measured.
• Sentiment with very little data is weak: SST-2 accuracy is 58% at 100 examples and 71% at 1,000.
• English only, cased 30k vocabulary, maximum 512 tokens. SQuAD contexts longer than the window are handled with a sliding window.
• CPU speed numbers are noisy and depend on the machine.
• Packed training sequences let attention cross document boundaries.
• Validation chunks come from the same source documents as training, so MLM-style perplexities from earlier stages are slightly optimistic.
• Not evaluated for bias, toxicity or factual reliability; it inherits the biases of web, encyclopedic, news and 19th/20th-century literary text.
• No evaluation yet on retrieval or sentence-embedding benchmarks.
• Custom code: loading needs trust_remote_code=True. Review the .py files in the repo before running them.

---

Development history

Stage	GLUE dev avg	What changed
v1 (MLM, ~1B tokens × 2 epochs)	75.0	Baseline
v2 (MLM, + 1.68B new tokens)	75.2	More data, lower MLM loss
V3 (ELECTRA), basic fine-tuning	76.5 – 77.3	Same style of basic recipe: +1.3 to +2.1 from the ELECTRA objective alone; RTE still near chance (52–54%)
+ MNLI transfer for RTE/MRPC/STS-B, layer-wise LR decay, more epochs, multi-seed	79.4	RTE rose from chance to ≈ 69%
+ <s>/</s> format matching pre-training	80.5	RTE 72.6, MRPC 87.4, STS-B 87.6, CoLA 52.3

---

Files, citation

Files: config.json, model.safetensors, configuration_Lewis_44M_Bert.py, modeling_Lewis_44M_Bert.py, tokenizer.json, tokenizer_config.json, special_tokens_map.json, plus (recommended) finetune_glue_final.py, results.json, seed_stats.json.

@misc{lewis44mbertv3,
  title  = {Lewis-44M-Bert-V3: a compact ELECTRA-style encoder pretrained from scratch on a single consumer GPU},
  author = {Viraj Salunke},
  year   = {2026},
  url    = {https://huggingface.co/vprojectx/Lewis-44M-Bert-V3}
}

References

• Clark et al., ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators, ICLR 2020. https://arxiv.org/abs/2003.10555
• Devlin et al., BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, NAACL 2019.
• Wang et al., GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding, ICLR 2019.
• Rajpurkar et al., SQuAD: 100,000+ Questions for Machine Comprehension of Text, EMNLP 2016.
• Tjong Kim Sang and De Meulder, Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition, 2003.