Update README.md

README.md

@@ -4,143 +4,3 @@ license: other
 license_name: cc-by-nc-4.0
 pipeline_tag: text-generation
 ---
-
-# Nemotron-Flash-3B Base Model
-
-<p align="center">
-  🗞️ <a href="https://arxiv.org/pdf/2511.18890">Paper</a>&nbsp&nbsp| &nbsp&nbsp 🤗 <a href="https://huggingface.co/nvidia/Nemotron-Flash-1B">Nemotron-Flash-1B</a> | &nbsp&nbsp 🤗 <a href="https://huggingface.co/nvidia/Nemotron-Flash-3B">Nemotron-Flash-3B</a> | &nbsp&nbsp 🤗 <a href="https://huggingface.co/nvidia/Nemotron-Flash-3B-Instruct">Nemotron-Flash-3B-Instruct</a> &nbsp
-</p>
-
-## Model Overview
-
-Nemotron-Flash is a new hybrid small language model family designed around real-world latency rather than parameter count. It features latency-optimal depth–width ratios, hybrid operators discovered through evolutionary search, and training-time weight normalization. See our <a href="https://arxiv.org/pdf/2511.18890">NeurIPS 2025 paper</a> for more technical details.
-
-The models achieve SOTA accuracy in math, coding, and commonsense reasoning at the 1B and 3B scales, while delivering decent small-batch latency and large-batch throughput. For example, Nemotron-Flash-1B achieves +5.5% accuracy, 1.9× lower latency, and 45.6× higher throughput compared with Qwen3-0.6B; and Nemotron-Flash-3B achieves +2% / +5.5% accuracy over Qwen2.5-3B / Qwen3-1.7B with 1.3× / 1.7× lower latency and 6.4× / 18.7× higher throughput, respectively.
-
-
-<div align="center">
-<img src="https://huggingface.co/nvidia/Nemotron-Flash-3B/resolve/main/images/nemotron_flash_result.png" alt="Compare with SOTA SLMs" width="800">
-</div>
-
-
-## Environment
-```bash
-torch<=2.9.1
-transformers<=4.56.2
-causal-conv1d
-flash-attn<=2.7.3
-mamba-ssm
-flash-linear-attention
-```
-We provide a <a href="https://huggingface.co/nvidia/Nemotron-Flash-3B/resolve/main/setup.sh">script</a> to build the conda environment: `bash setup.sh`.
-
-
-
-## Chat with Nemotron-Flash
-
-We integrated the attention kernel from <a href="https://nvidia.github.io/TensorRT-LLM/torch/auto_deploy/auto-deploy.html">TRT-LLM AutoDeploy</a> to enable generation with CUDA Graph:
-
-```
-from transformers import AutoModelForCausalLM, AutoTokenizer
-import torch
-
-repo_name = "nvidia/Nemotron-Flash-3B"
-
-tokenizer = AutoTokenizer.from_pretrained(repo_name, trust_remote_code=True)
-model = AutoModelForCausalLM.from_pretrained(repo_name, trust_remote_code=True)
-model = model.cuda().to(torch.bfloat16)
-
-print('Initializing generation state...')
-generation_state = model.init_cuda_graph_generation(
-    max_new_tokens=max_new_tokens,
-    batch_size=1,
-    device='cuda',
-)
-
-prompt = input("User:")
-inputs = tokenizer(prompt, return_tensors="pt").to('cuda')
-
-print(f"Generating with CUDA graph acceleration...")
-outputs = model.generate_with_cuda_graph(
-    input_ids=inputs["input_ids"],
-    generation_state=generation_state,
-    max_new_tokens=256,
-    temperature=0,
-    eos_token_id=tokenizer.eos_token_id,
-)
-
-response = tokenizer.decode(outputs[0][inputs['input_ids'].shape[1]:], skip_special_tokens=True)
-print(f"Response: {response}")
-```
-
-Another option is to perform generation w/o CUDA Graph:
-
-```
-outputs = model.generate_with_cache(
-    input_ids=inputs["input_ids"],
-    max_new_tokens=256,
-    temperature=0,
-    eos_token_id=tokenizer.eos_token_id,
-    )
-```
-
-## Finetune Nemotron-Flash
-
-To finetune Nemotron-Flash models, switch the attention kernel to FlashAttention2 when loading the model:
-
-```
-from transformers import AutoConfig, AutoModelForCausalLM
-repo_name = "nvidia/Nemotron-Flash-3B"
-
-config = AutoConfig.from_pretrained(repo_name, trust_remote_code=True)
-setattr(config, "attn_implementation_new", "flash_attention_2")
-model = AutoModelForCausalLM.from_pretrained(repo_name, config=config, torch_dtype=torch.bfloat16, trust_remote_code=True)
-```
-
-## Running Nemotron-Flash with TensorRT-LLM
-
-### Setup
-Installation + quick start for TensorRT-LLM: <a href="https://nvidia.github.io/TensorRT-LLM/quick-start-guide.html">Tutorial</a>.
-
-### Quick example
-
-An example script for running through the generation workflow:
-```
-cd examples/auto_deploy
-python build_and_run_ad.py --model nvidia/Nemotron-Flash-3B-Instruct --args.yaml-extra nemotron_flash.yaml
-```
-
-### Serving with trtllm-serve
-
-- Spin up a trtllm server (more details are in this <a href="https://nvidia.github.io/TensorRT-LLM/commands/trtllm-serve/trtllm-serve.html#starting-a-server">doc</a>):
-```
-trtllm-serve serve nvidia/Nemotron-Flash-3B-Instruct \
---backend _autodeploy \
---trust_remote_code \
---extra_llm_api_options examples/auto_deploy/nemotron_flash.yaml
-```
-
-- Send a request (more details are in this <a href="https://nvidia.github.io/TensorRT-LLM/examples/curl_chat_client.html">doc</a>):
-```
-curl http://localhost:8000/v1/chat/completions \
-    -H "Content-Type: application/json" \
-    -d '{
-        "model": "nvidia/Nemotron-Flash-3B-Instruct",
-        "messages":[{"role": "user", "content": "Where is New York?"}],
-        "max_tokens": 16,
-        "temperature": 0
-    }'
-```
-
-
-## Citation
-```
-@misc{fu2025nemotronflash,
-      title={Nemotron-Flash: Towards Latency-Optimal Hybrid Small Language Models}, 
-      author={Yonggan Fu and Xin Dong and Shizhe Diao and Matthijs Van keirsbilck and Hanrong Ye and Wonmin Byeon and Yashaswi Karnati and Lucas Liebenwein and Hannah Zhang and Nikolaus Binder and Maksim Khadkevich and Alexander Keller and Jan Kautz and Yingyan Celine Lin and Pavlo Molchanov},
-      year={2025},
-      eprint={2511.18890},
-      archivePrefix={arXiv},
-      primaryClass={cs.LG},
-      url={https://arxiv.org/abs/2511.18890}, 
-}