LLama3 SatCom 8B

LLama3 SatCom 8B is a fine-tuned open Large Language Model (LLM) developed under the ESA ARTES programme as part of the SatcomLLM / SCEVA (SatCom Expert Virtual Assistant) project.
It is designed to support satellite communications (SatCom) experts, engineers, and mission planners through domain-specialised reasoning, question answering, and document-based assistance.


Model Description

  • Base model: meta-llama/Llama-3.1-8B-Instruct
  • Fine-tuning type: Instruction fine-tuning (IFT)
  • Training data: Domain-specific question–answer datasets (manual, synthetic, and multiple-choice)
  • Architecture: Decoder-only transformer, 8 billion parameters
  • Languages: English
  • License: LLama-3.1 Communiy License Agreement

The model has been fine-tuned on curated SatCom-related corpora to enhance its understanding of technical language, protocols, and reasoning processes common to satellite communications, including 5G/6G non-terrestrial networks, link budget evaluation, and mission engineering tasks.


Training Datasets

Dataset Description
esa-sceva/satcom-synth-qa Synthetic QA data generated via agentic pipelines using large teacher models
esa-sceva/satcom-synth-qa-cot Chain-of-thought annotated QA used to improve reasoning depth and factual traceability

Intended Use

Primary use cases:

  • Technical Q&A and reasoning on SatCom systems
  • Support for link budget and RF engineering questions
  • Guidance for 5G/6G NTN (Non-Terrestrial Network) operations
  • Mission design, planning, and anomaly detection support
  • Educational and research use within the SatCom sector

Intended users:

  • ESA engineers and project officers
  • SatCom and aerospace researchers
  • SMEs and technical operators in satellite communication
  • Academic and educational users

Limitations

  • The model does not access real-time mission data or proprietary ESA documents.
  • Answers are based on training data and may require expert validation for operational use.
  • It should not be relied upon for flight-critical or safety-critical decisions.
  • Limited context window (base 8B configuration) may constrain long-document reasoning.

Technical Details

Parameter Value
Base Model Llama 3.1 8B Instruct
Parameters 8 billion
Context length 8k tokens
Precision bfloat16 / fp16
Framework Lit-GPT (Lightning AI)
Training infra EuroHPC MareNostrum5 + AWS EC2
Optimisation LoRA fine-tuning, cosine LR schedule

Evaluation

Evaluation Datasets

The model was benchmarked on both general-purpose and domain-specific QA tasks. Regarding Satcom-specific datasets:

Dataset Subset Description
esa-sceva/satcom-qa Open SatCom QA Conceptual and reasoning-based questions on SatCom workflows, regulations, and mission/system design
Math SatCom QA Quantitative and formula-based questions derived from system design and orbital mechanics topics
esa-sceva/satcom-mcqa Open MCQA Conceptual multiple-choice questions on RF systems, communication protocols, and architecture
Math MCQA Numerical and link-budget-focused multiple-choice questions testing applied calculations

Results

Model MCQA (Accuracy) Satcom-QA EVE-QA
Satcom EVE TeleQnA Norm. Bin. WR Norm. Bin. WR
Llama-3.1-8B-Instruct 78.59 81.35 68.40 75.44 70.40 61.47 65.20
llama3-satcom-8b 80.15 80.95 68.80 77.75 73.62 51.49 62.90 68.12 51.41

Table: Evaluation results (%).

  • Norm. denotes the normalized score obtained by averaging 1–5 ratings from a panel of LLM judges (Qwen3, gpt-4.1-mini, Mistral-Large-2512, and DeepSeek-V3.2) and scaling to [0,1].
  • Bin. is the binary accuracy computed from correctness judgments.
  • WR (Adjusted Win Rate) is defined as (wins + 0.5 × ties) / total, based on pairwise comparisons with randomized answer order.
    Multiple-choice performance is measured using standard accuracy.
    All results are averaged over 3–5 runs; standard deviation ≤ 0.25 pp for open-ended QA and ≤ 0.10 pp for MCQA.

Downloads last month
92
Safetensors
Model size
8B params
Tensor type
BF16
·
Inference Providers NEW
Input a message to start chatting with esa-sceva/llama3-satcom-8b.

Model tree for esa-sceva/llama3-satcom-8b

Finetuned
(2760)
this model

Datasets used to train esa-sceva/llama3-satcom-8b