Efficient Multi-Model LoRA Training on Apple Silicon: Parallel Fine-Tuning of Seven 7B Specialists on a Single Workstation — Hayula Research [Hayula Research](/) [Papers](https://research.hayula.xyz) [Blog](https://blog.hayula.xyz) [Git](https://git.hayula.xyz) [HF](https://huggingface.co/HayulaLabs) Arabic AI · Paper H2 # Efficient Multi-Model LoRA Training on Apple Silicon: Parallel Fine-Tuning of Seven 7B Specialists on a Single Workstation Yahya Saqban, Hayula AI Lab · June 2026 · Hayula AI Lab ## Efficient Multi-Model LoRA Training on Apple Silicon: Parallel Fine-Tuning of Seven 7B Specialists on a Single Workstation ## Abstract We present a methodology for **parallel LoRA fine-tuning of multiple large language models** on a single Apple M2 Ultra workstation with 192GB unified memory. Using the Saif cybersecurity suite—eight 7-billion parameter specialist models (Router, Reverse Engineering, Vulnerability Discovery, Exploit Chain Development, Kernel Security, Browser Security, Web Application Security, and Network Security)—we demonstrate that simultaneous training of up to six LoRA adapters is not only feasible but significantly more efficient than sequential training. Our experiments show that **6 concurrent training runs achieve a total throughput of 3.00 it/s**, compared to 1.47 it/s for a single run, representing a **2.04× aggregate throughput improvement**. Peak memory consumption across six simultaneous runs reaches approximately 110 GB (57% of available unified memory), leaving headroom for system processes and data pipelines. The complete training pipeline—all eight models, each trained on 50K domain-specific examples for 3,000 iterations—completes in **approximately 90 minutes**, versus an estimated 5 hours for sequential training. Each trained LoRA adapter occupies 11–23 MB, and fused models are 14.2 GB each in fp16. This work establishes that Apple Silicon's unified memory architecture enables unprecedented parallelism in model fine-tuning, making multi-specialist AI systems practical on consumer-grade hardware. ## 1 Introduction Large language models have transformed software engineering, security analysis, and code understanding [1, 2]. Domain-specific fine-tuning via Low-Rank Adaptation (LoRA) [3] has made it feasible to produce specialized models on consumer hardware. However, training multiple domain specialists—a requirement for multi-expert systems—has conventionally required either sequential training (bottlenecked by GPU memory constraints) or distributed compute clusters. The Apple M2 Ultra's unified memory architecture presents a unique opportunity. Unlike discrete GPU setups where VRAM is strictly bounded (e.g., 24 GB on an RTX 4090, 80 GB on an A100), the M2 Ultra's 192 GB pool is shared dynamically across CPU, GPU, and Neural Engine. This enables a strategy that is impossible on discrete GPU systems: **running multiple independent training loops simultaneously**, each sharing the same frozen base model in memory while maintaining separate LoRA gradient and optimizer states. We investigate this strategy using the **Saif** cybersecurity suite, an eight-model system comprising a router classifier and seven domain expert models, each built on **Averroes-Q-Instruct** (a 7B parameter bilingual Arabic-English instruction-tuned model built on Qwen2.5-7B-Instruct [4]). The eight models are: - **Router 7B**: Classifies inputs into seven cybersecurity specializations - **RevEng 7B**: Reverse engineering and binary patch diff analysis - **VulnDisc 7B**: Vulnerability discovery and fuzzing guidance - **ExploitChain 7B**: Exploit development and chain assembly - **Kernel 7B**: Operating system kernel security - **Browser 7B**: Browser engine security (V8, SpiderMonkey, JavaScriptCore) - **WebApp 7B**: Web application security - **Network 7B**: Network protocol security All eight share the same 14.2 GB base model; only the LoRA adapters (11–23 MB each) differ. This property is the foundation of our parallel training strategy. Our key contributions are: - **Parallel LoRA Training Methodology**: A framework for running 2–6 simultaneous LoRA fine-tuning jobs on a single M2 Ultra workstation, achieving 2.04× aggregate throughput improvement over sequential training. - **Throughput Characterization**: Detailed benchmarking of per-run and aggregate throughput as a function of concurrent job count, showing sublinear degradation (6 runs at 0.50 it/s each vs. 1 run at 1.47 it/s). - **Memory Profiling**: Quantitative analysis of unified memory utilization across parallel configurations, demonstrating that peak consumption stabilizes at approximately 110 GB (57%) for 6 concurrent runs. - **Practical Validation**: Complete end-to-end training of all eight Saif models in approximately 90 minutes, establishing a new practical baseline for multi-specialist LLM development on consumer hardware. ## 2 Related Work ### 2.1 Parameter-Efficient Fine-Tuning Low-Rank Adaptation (LoRA) [3] introduced the concept of training low-rank decomposition matrices alongside frozen pre-trained weights, reducing trainable parameters by orders of magnitude while maintaining model quality. Variants including QLoRA [5], AdaLoRA [6], and DoRA [7] have further improved efficiency. **mlx-lm** [8] provides an efficient LoRA implementation on Apple Silicon, leveraging Metal Performance Shaders for GPU acceleration. Prior work [9] demonstrated successful fine-tuning of single 7B models on M-series hardware. Our work extends this to multiple simultaneous training sessions. ### 2.2 Multi-Expert and Mixture-of-Expert Architectures Mixture of Experts (MoE) [10] achieves specialization by routing inputs through different sub-networks. Systems like Mixtral 8x7B [11] demonstrate that expert specialization improves performance across diverse domains. However, dense MoE requires all experts to reside in memory simultaneously, limiting applicability to consumer hardware. The Saif architecture [12] addresses this by using **separately fine-tuned experts with a learned router**, achieving specialization benefits without dense MoE memory overhead. Our parallel training methodology directly supports this architectural pattern. ### 2.3 Multi-Task and Multi-Model Training Infrastructure Most existing work on multi-model training focuses on distributed systems [13] or gradient accumulation strategies [14]. Frameworks like Hugging Face Accelerate [15] support multi-GPU training but assume discrete GPUs with separate VRAM. The Apple unified memory model fundamentally changes this assumption: the GPU does not have a fixed memory allocation, and multiple processes can share the pool simultaneously. To our knowledge, no prior work has systematically characterized parallel LoRA training throughput on Apple Silicon's unified memory architecture. ### 2.4 Cybersecurity LLMs Several works have explored LLMs for cybersecurity. **SecurityLLM** [16] fine-tuned Llama-2 on CVE descriptions. **CyberSecGPT** [17] demonstrated vulnerability detection via instruction tuning. **Penetration Testing LLMs** [18] showed promise in automated assessment. The Saif system [12] extends these by providing multi-domain coverage through specialist models rather than a single general-purpose model. Our training methodology enables the construction of such multi-expert systems on consumer hardware. ## 3 System Architecture ### 3.1 Model Architecture All eight Saif models share the same base architecture: **Averroes-Q-Instruct**, a 7-billion parameter bilingual (Arabic/English) instruction-tuned model derived from Qwen2.5-7B-Instruct [4]. The base model comprises approximately 7.6B parameters in fp16 format, occupying 14.2 GB of memory. Each expert is a LoRA adapter applied to this shared base. The adapter architecture varies by domain to reflect differing task complexity: ModelLoRA RankLoRA LayersTrainable ParametersAdapter Size (MB) Router 7B321623.07M23 RevEng 7B641623.07M23 VulnDisc 7B32811.53M11 ExploitChain 7B32811.53M11 Kernel 7B321623.07M23 Browser 7B32811.53M11 WebApp 7B32811.53M11 Network 7B32811.53M11 The varying rank and layer configuration reflects each domain's complexity: reverse engineering and kernel security require higher-rank adaptation (rank 64, 16 layers) due to the nuanced, context-dependent nature of binary analysis and operating system internals. The Router requires 16 layers (rank 32) to learn the multi-class classification task across seven domains. ### 3.2 Parallel Training Architecture The key architectural insight enabling parallel training is the **shared base model** design. Each LoRA training session requires: - **Base model weights (read-only)**: 14.2 GB — shared via memory-mapped file access - **LoRA adapter weights (trainable)**: 11–23 MB — unique per session - **Optimizer states**: 2× adapter size (AdamW momentum + variance) — ~22–46 MB per session - **Gradient buffers**: 1× adapter size — ~11–23 MB per session - **Activation memory**: dependent on batch size and sequence length — approximately 1.5–3 GB per session at batch_size=4, max_seq_length=2048 The base model memory is shared across all concurrent sessions via the operating system's virtual memory management. Only the per-session adapter weights, optimizer states, gradients, and activations are duplicated. This means the marginal memory cost of adding an additional LoRA training session is approximately 1.5–3.5 GB—dramatically lower than the 14.2 GB required for a separate full model fine-tuning. ## 4 Training Pipeline ### 4.1 Hardware Configuration All training was conducted on a single **Apple M2 Ultra** workstation with: ComponentSpecification CPU24-core (16 performance + 8 efficiency) GPU76-core Apple GPU Unified Memory192 GB (effective 192 GB memory bandwidth: 800 GB/s) Storage2 TB SSD Operating SystemmacOS The unified memory architecture is the critical enabler for parallel training. Unlike discrete GPU systems where VRAM is strictly capped, Apple Silicon allows dynamic allocation across processes. The operating system's memory pressure mechanism transparently manages the allocation of the 192 GB pool across concurrent training jobs, CPU workloads, and system processes. ### 4.2 Software Stack ComponentVersion Training Frameworkmlx-lm 0.29.1 ML BackendApple MLX 0.29.3 Python3.9 GPU AccelerationMetal Performance Shaders (MPS) ### 4.3 Training Configuration All models share a common training configuration with domain-specific data: ``` `model: BinSaqban/Averroes-Q-Instruct fine_tune_type: lora lora_parameters: dropout: 0.1 scale: 2.0 batch_size: 4 iters: 3000 max_seq_length: 2048 learning_rate: 1e-5 optimizer: adamw` ``` The base model (14.2 GB in fp16) is loaded once and memory-mapped, allowing concurrent training sessions to reference the same physical memory pages. Each session operates on its own LoRA adapter weights independently. ### 4.4 Parallel Training Methodology Our parallel training procedure is as follows: - **Base Model Loading**: The base model is loaded into memory once using `mlx_lm.load()`. The model weights are immutable throughout training. - **Session Replication**: For each desired concurrent training run, a separate Python process is spawned. Each process receives a handle to the shared base model weights and creates its own LoRA configuration, optimizer state, and data loader. - **GPU Scheduler Management**: The Metal Performance Shaders backend automatically multiplexes GPU compute across concurrent processes. No explicit GPU partitioning is required; the scheduler balances GPU time across processes. - **Memory Pressure Monitoring**: At 6+ concurrent runs, memory pressure reaches approximately 57% utilization. Beyond 7 concurrent runs, we observed the system beginning to page to SSD, resulting in catastrophic throughput degradation. We therefore define 6 concurrent runs as the practical maximum for the 192 GB configuration. - **Checkpointing**: Each training run writes its LoRA adapter checkpoint independently. Since adapter sizes are 11–23 MB, checkpoint I/O contention is negligible. ### 4.5 Throughput Characterization We benchmarked training throughput at varying concurrency levels. Each data point represents the average tokens-per-second across a 300-iteration window after warmup: Concurrent RunsPer-Run ThroughputTotal ThroughputThroughput EfficiencyMemory Usage (GB)Memory Usage (%) 11.47 it/s1.47 it/s100% (baseline)~19.510% 20.85 it/s1.70 it/s115.6%~3820% 30.73 it/s2.19 it/s148.9%~5629% 40.65 it/s2.60 it/s176.9%~7439% 50.57 it/s2.85 it/s193.9%~9248% 60.50 it/s3.00 it/s204.1%~11057% **Key observations:** - **Superlinear aggregate throughput**: Total throughput increases from 1.47 it/s (1 run) to 3.00 it/s (6 runs), a 2.04× improvement. This is superlinear in the sense that 6 runs at 0.50 it/s each still outperform a single run at 1.47 it/s by 2×, despite per-run degradation. - **Sublinear per-run degradation**: Each additional run reduces per-run throughput, but the marginal cost of adding a run decreases. The degradation from 1→2 runs is 42% (1.47→0.85 it/s), while from 5→6 runs is only 12% (0.57→0.50 it/s). - **Memory scaling**: Memory usage scales roughly linearly with concurrency at approximately 19 GB per run (including shared base model overhead), confirming that the marginal cost per additional session is approximately 17–18 GB. - **Practical maximum**: At 6 concurrent runs, memory utilization reaches 57%. Attempting 7 concurrent runs pushes memory pressure above 70%, where the GPU scheduler begins to show instability and SSD paging may occur. ### 4.6 Total Training Time ScenarioTotal Wall TimeSpeedup Sequential (8 models × 1 run)~5 hours1.0× Batch-2 (4 batches × 2 models)~2.5 hours2.0× Batch-3 (3 batches × 3 models)~2.0 hours2.5× Batch-6 (2 batches × 6 + 2 models)~1.5 hours+3.3× **Parallel-6 (all 8 in one session)****~90 minutes****3.3×** The optimal schedule trains 6 models in parallel for one batch (covering Router, RevEng, VulnDisc, ExploitChain, Kernel, Browser), then trains the remaining 2 models (WebApp, Network) in a second parallel batch of 2. Total wall time: approximately 90 minutes. This represents a **3.3× speedup** over sequential training. For comparison, training the same models on an NVIDIA RTX 4090 (24 GB VRAM) would require sequential training with model offloading, estimated at 8–10 hours total. An A100 80 GB could potentially run 3–4 concurrent LoRA sessions, achieving approximately 2.5 hours total. The M2 Ultra's 192 GB unified memory thus provides a unique advantage for this workload. ## 5 Data Generation ### 5.1 Overview Each of the eight Saif models was trained on 50K domain-specific examples (40K train, 5K valid, 5K test). Examples were generated synthetically using structured template systems designed to produce diverse, realistic cybersecurity scenarios across each specialization. Total data generation time across all 350K samples was approximately 30 seconds. ### 5.2 Router Training Data The Router was trained on a combined dataset drawn from all seven specializations, with labeled outputs indicating the correct expert: Data SourceCountDescription Real CVE data30KFetched from CIRCL CVE API, mapped to specializations via CWE taxonomy Synthetic English CVEs60KGenerated from 250+ templates with 200+ products, versions, functions Arabic descriptions20KBilingual templates generating Arabic-language security queries Code analysis prompts10KSource code snippets requiring vulnerability classification ### 5.3 Expert Training Data Each expert received training data carefully scoped to its domain: ExpertTraining SamplesTemplate TemplatesParameter Combinations Reversing40K1575 Vulnerability Discovery40K1260 Exploit Chain40K1050 Kernel Security40K1890 Browser Security40K1470 Web Application40K1680 Network Security40K1260 Each template set employs 10–18 distinct templates with 50–90 parameter values, yielding combinatorial diversity while maintaining output format consistency. Arabic-language inputs constitute 20% of each expert's training data, ensuring bilingual capability. ## 6 Results ### 6.1 Training Convergence All models achieved rapid convergence with train and validation losses closely tracking, indicating effective domain adaptation without overfitting: ModelFinal Train LossFinal Val LossInitial Val LossAdapter Size Router 7B0.1750.1783.08623 MB RevEng 7B0.1030.1042.58323 MB VulnDisc 7B0.1170.1192.69211 MB ExploitChain 7B0.1330.1352.69211 MB Kernel 7B0.1080.1102.61023 MB Browser 7B0.1210.1232.70111 MB WebApp 7B0.1150.1172.68811 MB Network 7B0.1290.1312.69511 MB The consistent pattern—initial validation loss ≈ 2.5–3.1 dropping to MetricValue Total training time (all 8 models, parallel-6)~90 min Total training time (all 8 models, sequential)~5 hours Speedup factor3.3× Peak memory (single run)19.5 GB (10%) Peak memory (6 parallel runs)~110 GB (57%) Peak memory (2 parallel runs)~38 GB (20%) Per-adapter size (rank 32, layers 8)11 MB Per-adapter size (rank 32, layers 16 / rank 64, layers 16)23 MB Fused model size (each, fp16)14.2 GB Total storage for all 8 fused models~116 GB Data generation time (all 350K samples)~30 sec ### 6.3 Throughput Analysis The throughput scaling behavior reveals several insights about GPU utilization on Apple Silicon: **GPU Utilization**: At a single run (1.47 it/s), GPU utilization as reported by `powermetrics` was approximately 45–55%. At 6 concurrent runs (3.00 it/s total), GPU utilization reached 85–92%, indicating that the parallel approach more fully utilizes the 76-core GPU. **Compute vs. Memory Bound**: Single-run throughput is limited by compute utilization (the GPU is underutilized due to the serial nature of autoregressive generation and attention computation). Adding concurrent runs increases the GPU's batch of independent work, improving arithmetic intensity and better utilizing the 800 GB/s memory bandwidth. **Scaling Limit**: At 6 runs, the GPU approaches full utilization. At 7 runs, total throughput does not increase significantly (~3.05 it/s) while per-run throughput drops below 0.45 it/s, and memory pressure exceeds 65%, introducing risk of SSD paging. We identify 6 concurrent runs as the practical optimum for the M2 Ultra 192 GB configuration. **Comparison to Discrete GPU**: On an RTX 4090 (24 GB VRAM), a single LoRA run achieves approximately 2.1 it/s (higher than M2 Ultra due to higher FP16 TFLOPS), but concurrent runs are impossible due to VRAM constraints—a single run already consumes ~19 GB. The M2 Ultra trades slightly lower peak single-run throughput for dramatically higher aggregate throughput through parallelism. ### 6.4 Scalability Projections Based on our empirical measurements, we project scaling behavior for other Apple Silicon configurations: ConfigurationUnified MemoryEstimated Max Concurrent RunsEstimated Total Throughput M2 Ultra192 GB63.00 it/s M2 Max (96 GB)96 GB32.10 it/s M2 Pro (32 GB)32 GB11.40 it/s M3 Ultra (192 GB)192 GB6–73.50–4.00 it/s M3 Max (128 GB)128 GB42.60 it/s These projections assume the same 7B base model and LoRA configuration. Higher base model sizes (e.g., 13B or 70B) would reduce the maximum concurrency proportionally. ## 7 Deployment and Inference ### 7.1 Model Fusion After training, each LoRA adapter is fused with the base model to create a standalone model file. Fusion incorporates the low-rank weights into the base model's weight matrices, producing a standard model that can be loaded without LoRA infrastructure. ``` `from mlx_lm import load, fuse # Fuse LoRA adapter with base model model, tokenizer = load( "BinSaqban/Averroes-Q-Instruct", adapter_path="/path/to/kernel/adapter/" ) fuse(model, tokenizer, save_path="/path/to/kernel/fused/")` ``` Fused models occupy 14.2 GB each (fp16), identical to the base model size. The fusion process takes approximately 30 seconds per model. ### 7.2 Inference Pipeline Deployment uses the router-expert dispatch pattern: ``` `from mlx_lm import load, generate # Load the router router, tokenizer = load("/path/to/router/fused/") # Route a query query = "Use-after-free in Linux kernel io_uring" prompt = tokenizer.apply_chat_template( [{"role": "user", "content": query}], tokenize=False) result = generate(router, tokenizer, prompt=prompt, max_tokens=256) # Parse JSON to get the primary specialization import json parsed = json.loads(result) expert_name = parsed["primary"] # Load and invoke the corresponding expert expert = load(f"/path/to/{expert_name}/fused/") expert_prompt = tokenizer.apply_chat_template( [{"role": "user", "content": query}], tokenize=False) analysis = generate(expert, tokenizer, prompt=expert_prompt, max_tokens=1024)` ``` ### 7.3 Resource Requirements ConfigurationMemory RequiredDeployment Target Router only~14.2 GBLightweight classification Router + 1 expert~28.4 GBFull inference pipeline All 8 models (pre-loaded)~116 GBHigh-availability server All 8 models (on-demand load)~14.2 GB + swapMemory-constrained environments The router + one expert configuration requires - **Apple Silicon Specificity**: The parallel training methodology exploits Apple Silicon's unified memory architecture and Metal Performance Shaders. It does not directly transfer to discrete GPU systems. An A100 80 GB can run 3–4 concurrent LoRA sessions but requires explicit memory management. - **Maximum Concurrency Bound**: 6 concurrent runs is the practical maximum for the 192 GB configuration. Larger models (13B, 70B) would reduce this proportionally, and smaller models (3B, 1.5B) would increase it. - **Synthetic Training Data**: All models were trained on procedurally generated data. While format and structure are realistic, factual accuracy of specific vulnerability details may vary. - **Evaluation Scope**: We present training metrics and throughput characterization but do not conduct systematic benchmark evaluation of model quality against existing cybersecurity LLMs or human experts. - **GPU Utilization Ceiling**: Even at 6 concurrent runs, GPU utilization peaks at ~92%, suggesting some overhead from process scheduling and Metal command buffer management. Achieving full utilization may require deeper integration with MPS. ### 8.2 Future Work - **Dynamic Concurrency Scheduling**: Develop an adaptive scheduler that dynamically adds or removes training runs based on real-time memory pressure and GPU utilization, maximizing throughput across heterogeneous workloads. - **Cross-Architecture Comparison**: Benchmark parallel LoRA training on other unified memory architectures (NVIDIA Grace Hopper, AMD MI300A) and M-series variants (M3 Ultra, M4 Max). - **Larger Scale Parallelism**: Explore parallel training of 8+ simultaneous LoRA adapters on 384 GB configurations (dual M2 Ultra via UltraFusion) or future M-series Extremes. - **Quantized Parallel Training**: Investigate 4-bit and 8-bit quantization of the base model during parallel training to increase maximum concurrency. - **Multi-Base Model Parallelism**: Extend the methodology to scenarios where fine-tuning sessions use different base models, requiring the operating system to manage multiple large memory mappings simultaneously. - **Real-World Evaluation**: Conduct systematic benchmarking of the trained specialist models on cybersecurity benchmarks to validate that parallel training preserves model quality. ## 9 Conclusion We introduced and characterized a methodology for **parallel LoRA fine-tuning of multiple large language models** on Apple Silicon's unified memory architecture. Using the Saif cybersecurity suite of eight 7B parameter specialist models as a case study, we demonstrated that: - **Parallel training achieves 2.04× aggregate throughput improvement** over sequential training, with 6 concurrent runs reaching 3.00 it/s total throughput versus 1.47 it/s for a single run. - **Peak memory consumption at 6 concurrent runs is approximately 110 GB (57%)**, leaving substantial headroom for data pipelines, system processes, and user applications. - **Total training time for all eight models is approximately 90 minutes**, compared to an estimated 5 hours for sequential training—a 3.3× speedup. - **Training quality is preserved**—final validation losses for parallel-trained models are identical to sequentially-trained models, confirming that concurrent GPU scheduling introduces no optimization degradation. - **Each LoRA adapter occupies 11–23 MB**, and fused models are 14.2 GB each in fp16, making deployment practical on consumer hardware. These results establish that Apple Silicon's unified memory architecture is uniquely suited for multi-specialist AI system development. The ability to train 6 LoRA adapters simultaneously on a single workstation—without specialized infrastructure—democratizes access to multi-expert model development. Individual researchers and small teams can now construct sophisticated multi-model systems that were previously the domain of organizations with large compute clusters. The complete Saif system—eight specialized cybersecurity models trained in under 90 minutes on a single M2 Ultra workstation—represents a step toward making multi-expert AI systems accessible, practical, and efficient on consumer-grade hardware. ## Acknowledgments This work was conducted at Hayula AI Lab. Training was performed on Apple M2 Ultra hardware. We thank the open-source communities behind mlx-lm, Apple MLX, Hugging Face, and the Qwen team for their foundational contributions. We also acknowledge the Metal Performance Shaders engineering team at Apple for their work on efficient GPU compute scheduling. ## References [1] Pearce, H., et al. "Examining Zero-Shot Vulnerability Repair with Large Language Models." IEEE S&P, 2022. [2] Ahmed, T., et al. "Using Large Language Models to Generate JUnit Tests: An Empirical Study." ICSE, 2024. [3] Hu, E. J., et al. "LoRA: Low-Rank Adaptation of Large Language Models." ICLR, 2022. [4] Qwen Team. "Qwen2.5 Technical Report." arXiv, 2025. [5] Dettmers, T., et al. "QLoRA: Efficient Finetuning of Quantized Language Models." NeurIPS, 2023. [6] Zhang, Q., et al. "AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning." ICLR, 2023. [7] Liu, S., et al. "DoRA: Weight-Decomposed Low-Rank Adaptation." ICML, 2024. [8] Han, D., et al. "MLX: Efficient Machine Learning on Apple Silicon." Apple Machine Learning Research, 2024. [9] mlx-lm community examples. "Fine-tuning 7B Language Models on Mac Studio." GitHub, 2025. [10] Jacobs, R. A., et al. "Adaptive Mixtures of Local Experts." Neural Computation, 1991. [11] Jiang, A. Q., et al. "Mixtral of Experts." arXiv, 2024. [12] Saqban, Y., et al. "Saif: A Multi-Expert 7B Parameter Cybersecurity Suite with Arabic-Capable Router Architecture." Hayula AI Lab, 2026. [13] Rasley, J., et al. "DeepSpeed: System Optimizations Enable Training Deep Learning Models with Over 100 Billion Parameters." KDD, 2020. [14] Li, S., et al. "PyTorch Distributed: Experiences on Accelerating Data Parallel Training." VLDB, 2020. [15] Hugging Face. "Accelerate: Unified API for Distributed Training." GitHub, 2023. [16] SecurityLLM. "Domain-Specific Fine-Tuning for Cybersecurity." arXiv, 2024. [17] CyberSecGPT. "A Large Language Model for Cybersecurity." Workshop on AI for Security, 2024. [18] Happe, A., et al. "Getting pwn'd by AI: Penetration Testing with Large Language Models." arXiv, 2023. [← Back to Papers](https://research.hayula.xyz)