techkiyan/KiyanNet-Finance-Base-48M

KiyanNet Finance Base 48M

KiyanNet Finance Base 48M is a compact finance-domain causal language model published in standard Hugging Face Transformers format. It is intended as a lightweight baseline for finance and India-market language modeling, continued pretraining, and downstream fine-tuning.

This is a base model, not an instruction-tuned assistant. It should not be used to provide buy/sell calls, price targets, guaranteed outcomes, or personalized investment advice.

Load With Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "techkiyan/KiyanNet-Finance-Base-48M"

tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id)

prompt = "Reliance Industries reported"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(
    **inputs,
    max_new_tokens=80,
    do_sample=True,
    temperature=0.8,
    top_p=0.95,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

No custom Python files are required.

Files

This public repo intentionally contains only standard loading and fine-tuning files:

• config.json
• generation_config.json
• model.safetensors
• tokenizer.json
• tokenizer_config.json
• README.md

Training scripts, logs, token shards, raw datasets, duplicate checkpoints, and custom model-code files are intentionally excluded.

Model Details

Item	Value
Public loader	AutoModelForCausalLM
Model class	GPT2LMHeadModel
Tokenizer loader	AutoTokenizer
Tokenizer type	Fast BPE tokenizer converted from original SentencePiece model
Original trained parameters	48,376,320
HF model parameters	48,433,152
Context length	1,024
Vocabulary size	32,000
Layers	10
Attention heads	8
Hidden size	512
Training tokens processed	606,601,216
Training steps	9,256
Original best validation loss	2.6867
Original best validation perplexity	14.68

The HF parameter count is slightly higher because the public Transformers conversion uses GPT-2 modules with zero-initialized bias tensors. The converted model was checked against the original checkpoint; logits matched within normal floating-point tolerance on a fixed prompt.

Fine-Tuning Starter

from transformers import AutoModelForCausalLM, AutoTokenizer, Trainer, TrainingArguments

repo_id = "techkiyan/KiyanNet-Finance-Base-48M"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id)

if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

# Prepare a tokenized causal-LM dataset before passing it to Trainer.
training_args = TrainingArguments(
    output_dir="kiyannet-finetuned",
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,
    learning_rate=5e-5,
    num_train_epochs=1,
    logging_steps=25,
    save_steps=500,
)

Benchmark Protocol

Benchmarks were run locally on CPU with torch==2.13.0.

The benchmark is intentionally finance-focused and lightweight. It is not a leaderboard result. It was designed to compare small standalone causal language models in the 30M-100M range under identical local conditions.

Model Selection

The target comparison set was 10 standalone AutoModelForCausalLM models between roughly 30M and 100M parameters.

Public finance-specific causal LMs in this exact size range are scarce. One finance-named 70M candidate, Rudra501/model70m_finance, appeared to be an adapter/training artifact rather than a standalone loadable checkpoint, so it was excluded. Larger finance models, GGUF-only models, classifiers, fill-mask models, and models above 100M parameters were also excluded.

Evaluation Methods

The benchmark uses multiple strategies:

• Category perplexity on finance-style snippets
• 20 fixed generation prompts across company, regulatory, technical-analysis, options, filing, and safety categories
• Greedy generation throughput
• Sampled generation throughput
• Repetition heuristic
• Bad-advice phrase heuristic
• Metadata-marker heuristic
• RBI role-error heuristic
• Nifty foreign-context heuristic
• Finance-tokenizer round-trip tests

10-Model Benchmark Summary

Lower perplexity is better. Higher tokens/sec is better.

Model	Params	Overall PPL ↓	Company PPL ↓	Regulatory PPL ↓	Technical PPL ↓	Options PPL ↓	Filing PPL ↓	Greedy tok/s ↑	Sample tok/s ↑	Round-trip
KiyanNet Finance Base 48M	48.4M	147.97	51.36	77.60	229.80	574.70	76.96	285.6	223.1	12/12
DistilGPT2 82M	81.9M	303.18	223.11	187.41	721.56	482.52	158.42	197.0	145.9	12/12
Pythia 31M	30.5M	521.45	671.44	411.97	846.98	670.09	252.87	704.4	302.8	12/12
Pythia 31M Deduped	30.5M	608.51	745.83	489.15	1313.32	660.85	286.13	709.9	302.1	12/12
Pythia 70M	70.4M	449.69	615.45	351.05	799.07	552.51	204.50	478.5	250.2	12/12
Pythia 70M Deduped	70.4M	402.49	552.47	265.86	797.37	513.86	182.32	475.7	249.9	12/12
Pythia 70M v0	70.4M	386.08	538.82	260.89	728.45	468.03	187.98	482.6	248.5	12/12
Pythia 70M Deduped v0	70.4M	340.24	425.04	288.84	543.95	404.98	176.49	479.2	249.5	12/12
Pythia 70M ChatSalad	70.4M	4825.02	18111.99	6785.18	4109.66	4577.80	1595.80	479.8	246.6	12/12
TinyStories 33M	68.5M	137002.50	313508.09	160010.96	168002.60	63643.98	128294.34	232.5	166.6	12/12

Diagnostic Heuristics

Model	Avg repetition ↓	Bad-advice hits ↓	Metadata-marker hits ↓	RBI role errors ↓	Nifty foreign-context hits ↓
KiyanNet Finance Base 48M	0.1664	1	0	0	0
DistilGPT2 82M	0.0508	1	0	0	0
Pythia 31M	0.2651	1	0	0	0
Pythia 31M Deduped	0.4137	1	0	0	0
Pythia 70M	0.2753	1	0	0	0
Pythia 70M Deduped	0.3282	1	0	0	0
Pythia 70M v0	0.2519	1	0	0	0
Pythia 70M Deduped v0	0.1641	1	0	0	0
Pythia 70M ChatSalad	0.0155	1	0	0	0
TinyStories 33M	0.0017	1	0	0	0

The bad-advice heuristic is intentionally broad and counts appearances of words such as buy, sell, target price, stop loss, guaranteed, and risk-free. It is useful for smoke testing but should not be treated as a complete safety evaluation.

Prompt Coverage

The generation benchmark used 20 prompts:

• Reliance Industries reported
• Tata Motors quarterly results showed
• HDFC Bank announced
• Infosys management stated
• State Bank of India reported
• The Reserve Bank of India announced
• SEBI issued a circular regarding
• NSE Clearing informed members that
• Nifty 50 approached resistance while
• Bank Nifty tested support near
• RSI showed bearish divergence because
• Volume did not confirm the breakout
• IV rank differs from IV percentile because
• Rising open interest with falling futures prices indicates
• An iron condor has maximum loss when
• Put-call ratio should not be interpreted alone because
• The company announced a board meeting to consider
• The rights issue will dilute shareholders if
• The IPO risk factors indicate
• This market summary should not provide a buy or sell recommendation because

Tokenizer Round-Trip Checks

The current tokenizer preserves finance notation in all tested examples:

Text	Round-trip
₹1.06 crore	pass
₹3,642.50	pass
7.2%	pass
FY2025-26	pass
NIFTY 50	pass
BANKNIFTY	pass
USD/INR	pass
P/E	pass
EV/EBITDA	pass
F&O	pass
52-week high	pass
25-delta risk reversal	pass

Sample KiyanNet Outputs

Greedy decoding examples from the benchmark:

Prompt	Output excerpt
Reliance Industries reported	Reliance Industries reported a net loss of 1 5 billion in the third quarter of 2016...
Tata Motors quarterly results showed	Tata Motors quarterly results showed a strong performance in the first quarter of fiscal 2019...
HDFC Bank announced	HDFC Bank announced a new loan agreement with the Bank of England...
SEBI issued a circular regarding	SEBI issued a circular regarding the names of the concerned stock exchange(s) where the Equity Shares are proposed to be listed...
NSE Clearing informed members that	NSE Clearing informed members that such multiple settlements have been scheduled on Mar 22, 2019...

These samples show both strengths and weaknesses. The model has finance-domain fluency and performs well on the small local perplexity benchmark, but it still invents figures, mixes jurisdictions, and can reuse generic news or filing patterns. It should be treated as a raw-mix baseline, not a final India-first analyst model.

Interpretation

KiyanNet Finance Base 48M performs best on this local 30M-100M finance-smoke benchmark by overall perplexity, company-text perplexity, regulatory-text perplexity, technical-text perplexity, and filing-text perplexity. It is also faster than DistilGPT2 while using fewer parameters.

The main weakness is options and derivatives language. DistilGPT2 and Pythia 70M Deduped v0 scored better on the options subset. This supports the next training direction: add explicit options, futures, volatility, and technical-market-state data rather than continuing only with generic financial news.

Intended Use

• Compact finance-domain language modeling experiments
• Continued pretraining
• Fine-tuning for structured finance tasks
• Lightweight local baselines
• Financial text understanding research

Not Intended For

• Buy or sell recommendations
• Entry/exit levels
• Price targets
• Guaranteed return claims
• Personalized investment advice
• Live factual claims without retrieval or external verification
• Option payoff or Greek calculation without deterministic tools

Limitations

• This is a base model, not an instruction-tuned analyst.
• It can hallucinate dates, figures, institutions, currencies, and corporate actions.
• It can mix India-specific prompts with global-news templates.
• Options and derivatives understanding is still weak.
• Generated text should be verified against reliable data sources.
• The benchmark above is a small local smoke benchmark, not a broad public leaderboard.

Recommended Next Step

Preserve this model as a 607M-token raw-mix baseline. The next stronger base model should use a balanced India-first training mix with explicit regulatory, corporate-announcement, technical-analysis, options, futures, volatility, and market-state data.