| --- |
| license: apache-2.0 |
| datasets: |
| - Allanatrix/Scientific_Research_Tokenized |
| language: |
| - en |
| base_model: |
| - Allanatrix/NexaSci |
| pipeline_tag: text-generation |
| tags: |
| - Science |
| - Hypothesis |
| - Methodology |
| --- |
| |
| # NexaSci Family of Models |
|
|
| ## Welcome to the NexaSci Repository! |
|
|
| Get ready to supercharge your scientific research with the **Nexasci family of models**! This Hugging Face repository hosts a powerful suite of Mixture-of-Experts (MoE) models designed to generate hypotheses and methodologies across **physics**, **biology**, and **materials science**. Built with efficiency and scalability in mind, the NexaSci family includes the baseline **NexaSci**, the reasoning-enhanced **NEXASci-1-CoT**, and the long-context powerhouse **NEXA-1-Max**. Whether you’re a researcher tackling complex STEM problems, a data scientist exploring scientific ML, or a student learning about domain-specific AI, this repository is your go-to resource for cutting-edge scientific computation. |
|
|
| ## Model Overview |
|
|
| The NexaSci family is a 110 million to 2.2 billion parameter architecture that uses a **Semantic Router** to direct queries to domain-specific expert modules (Physics, Biology, Materials Science). It’s optimized for resource-constrained environments, leveraging advanced training strategies, hardware optimizations, and techniques like reinforcement learning and sparse attention. Below are the current and planned models: |
|
|
| ### 1. NexaSci-1-Mini (Still working on this Indefinite timeline) |
| - **Parameters**: ~110 million |
| - **Purpose**: Generates hypotheses and methodological scaffolding for scientific tasks in physics, biology, and materials science. |
| - **Architecture**: |
| - **Semantic Router**: BERT-based classifier routes queries to domain-specific experts. |
| - **Expert Modules**: T5-based submodules for Physics, Biology, and Materials Science. |
| - **Inference & Validation Pipeline**: Aggregates expert outputs and ensures consistency. |
| - **Knowledge Feedback Loop**: Refines routing using reinforcement learning. |
| - **Training**: |
| - Pretrained on ~2B tokens from arXiv, PubMed, and other scientific corpora. |
| - Fine-tuned with QLoRA on 500k instruction-style samples. |
| - Uses AzureSky Optimizer (Stochastic Approximation + Adam hybrid). |
| - **Use Cases**: |
| - Generate plausible hypotheses (e.g., new material properties). |
| - Suggest experimental methods (e.g., protein folding protocols). |
| - Summarize scientific texts with domain-specific insights. |
|
|
| ### 2. NEXASci-1-COT (Coming Soon) |
| - **Parameters**: 756 million to 1.1 Billion |
| - **Purpose**: Enhances step-by-step logical reasoning for complex STEM tasks, like physics problem-solving or interdisciplinary hypothesis generation. |
| - **Architecture**: |
| - Adds a **Chain of Thought (CoT) Processor** with sparse attention (Longformer-style) for multi-step reasoning. |
| - Includes **Conditional Routing** to engage the CoT Processor based on a “reasoning_required” flag. |
| - Integrates with expert modules for structured, logical outputs. |
| - **Training**: |
| - Trained in three stages: Easy (basic logic), Moderate (complex tasks), Hard (advanced reasoning). |
| - Uses ~2B tokens |
| - Employs AzureSky Optimizer with reinforcement learning fine-tuning. |
| - **Use Cases**: |
| - Solve multi-step physics problems (e.g., astrophysics simulations). |
| - Generate detailed, logical methodologies (e.g., combining CFD and alloy modeling). |
| - Teach scientific reasoning in educational settings. |
| |
| ### 3. NEXASci-1-Max (Coming soon) |
| - **Parameters**: ~2.2 billion |
| - **Purpose**: Processes large scientific documents (up to 20,000 tokens) with deep contextual understanding. |
| - **Architecture**: |
| - Features a **Long Context Attention Layer** with two Flash Attention v2 layers for efficient long-sequence processing. |
| - Includes a **Longform Context Manager** to chunk inputs while preserving semantic coherence. |
| - Scales parameters using mixed precision training and gradient checkpointing. |
| - **Training**: |
| - Trained on ~2B tokens, including a Long-Context Corpus of full arXiv papers and NIH grants. |
| - Uses AzureSky Optimizer with mixed precision (FP16/BF16) and gradient checkpointing. |
| - **Use Cases**: |
| - Summarize or analyze long scientific papers (e.g., 120K-token preprints). |
| - Generate hypotheses from extended contexts (e.g., patent methods). |
| - Support multi-query tasks requiring deep document understanding. |
| |
| ### Future Models (Planned) |
| - **NEXASci-1-Scout**: A lightweight version (~50M parameters) optimized for distilling and curating datasets and maaking the corpa for the model family |
| - **NEXASci-1-Super**: A larger-scale model (~10B parameters) for advanced scientific tasks, using ~1B tokens. Planned for high-performance computing clusters. |
| - **NEXASci-1-MultiModal**: Integrates text, images, and graphs for scientific data analysis (e.g., protein structures, simulation plots). Planned for future research. |
| |
| ## Dataset and Training Details |
| |
| The NexaSci family is trained on a **tiered token strategy** to maximize efficiency and domain specificity, as outlined in the architecture document: |
| |
| - **Warm Start Corpus** (100M tokens): General language understanding from FineWeb-Edu, OpenWebMath, Wikipedia, and Aristo Science Questions. |
| - **Scientific Pretraining Corpus** (1-2B tokens): Domain-specific data from arXiv (physics), PubMed/BioRxiv (biology), and Materials Project/ChemRxiv (materials science). |
| - **Instruction Fine-Tune Dataset** (500K tokens): 5k high-quality instruction-style samples for hypothesis and method generation. |
| |
| **Token Efficiency Strategies**: |
| - Entropy scoring to remove low-information samples. |
| - Semantic tagging (e.g., [PHYS], [BIO], [MTH]) for domain routing. |
| - Distillation using larger models (e.g., GPT-4) to summarize and structure data. |
| - Routing and filtering to activate only relevant expert paths. |
| |
| **Total Token Budget**: |
| For all models ~2B tokens |
| |
| **Hardware**: |
| Currently limited here still looking and hunting |
| |
| **Optimization Techniques**: |
| - Sparse attention, mixed precision training, gradient checkpointing. |
| - Hyperparameter tuning with Optuna, Just-in-Time (JIT) compilation, multi-threading. |
| - AzureSky Optimizer for efficient convergence. |
| |
| |
| # Download Models: |
| |
| Model weights are hosted on Hugging Face. Download them using the transformers library or directly from the repository’s model card. |
| Example:huggingface-cli download your-username/nexamoe-base |
| |
| |
| # Usage |
| |
| Load a Model: Use the transformers library to load NexaMOE models: |
| ``` |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model_name = "your-username/nexasci-base" |
| tokenizer = AutoTokenizer.from_pretrained(model_name) |
| model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto") |
| |
| |
| Generate Hypotheses or Methods:Provide a prompt with optional domain tags: |
| prompt = "[PHYS] Suggest a hypothesis for dark matter detection." |
| inputs = tokenizer(prompt, return_tensors="pt").to("cuda") |
| outputs = model.generate(**inputs, max_length=200) |
| print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
| |
| |
| Use NEXA-CoT for Reasoning:Enable the CoT Processor for step-by-step logic: |
| prompt = "[BIO] [reasoning_required] Propose a method to predict protein folding." |
| inputs = tokenizer(prompt, return_tensors="pt").to("cuda") |
| outputs = model.generate(**inputs, max_length=500) |
| print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
| |
| |
| Process Long Documents with NEXA-Ultramax:Handle large inputs (up to 20,000 tokens): |
| with open("arxiv_paper.txt", "r") as f: |
| document = f.read() |
| prompt = f"[MAT] Summarize this document: {document}" |
| inputs = tokenizer(prompt, return_tensors="pt", truncation=True, max_length=20000).to("cuda") |
| outputs = model.generate(**inputs, max_length=1000) |
| print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
| |
| |
| Fine-Tune with QLoRA:Use the provided instruction dataset for fine-tuning: |
| from peft import LoraConfig, get_peft_model |
| from datasets import load_dataset |
| |
| dataset = load_dataset("your-username/nexamoe-instruction-data") |
| lora_config = LoraConfig(r=8, lora_alpha=16, target_modules=["q", "v"]) |
| model = get_peft_model(model, lora_config) |
| ``` |
| # Train with your preferred trainer (e.g., Hugging Face Trainer) |
| |
| Run Inference via CLI or GUI: |
| |
| "Command-Line: python inference.py --model your-username/nexamoe-base --prompt "[PHYS] Hypothesise a new superconductor." |
| |
| Opens a web interface to interact with the model. |
| |
| # Performance Metrics |
| |
| Extreme Specialisation: Modular experts improve response fidelity and interpretability. |
| Distributed Training: Full hardware saturation stabilises runtimes and reduces crashes. |
| Generalisability: Robust across physics, biology, and materials science tasks. |
| Optimiser Efficiency: AzureSky Optimiser enhances convergence speed and precision. |
| |
| See the architecture document for detailed loss curves and metrics. |
| Similar Models |
| Explore related models for inspiration: |
| |
| Grok (xAI): General-purpose conversational AI with scientific capabilities. Link |
| LLaMA (Meta AI): Efficient research models for NLP tasks. Link |
| SciBERT: BERT variant for scientific text processing. Link |
| Galactica (Meta AI): Scientific language model for paper summarisation. Link |
| BioBERT: BERT variant for biomedical text. Link |
| |
| For the models, cite: |
| Allanatrix. (2025). NexaMOE Family of Models. Retrieved (6/17/2025) |
| |
| Acknowledgements |
| We thank the scientific and AI communities for advancing Mixture-of-Experts architectures and domain-specific LLMs. Special thanks to the authors of the datasets used (arXiv, PubMed, Materials Project) and the developers of tools like Transformers, PEFT, and Optuna. |
| For more information, see https://materialsproject.org/, https://arxiv.org/, https://pubmed.ncbi.nlm.nih.gov/ |
| |
| License |
| MIT License (see the LICENSE file for details). |
| |
| Have questions or ideas? Open an issue on GitHub or join the discussion on Hugging Face. Happy researching! |
| |
