| |
|
| | --- |
| | license: mit |
| | tags: |
| | - time-series |
| | - mixture-of-experts |
| | - forecasting |
| | - pytorch |
| | - fft |
| | model-index: |
| | - name: SuperLinear |
| | results: [] |
| | --- |
| | |
| |
|
| | # Super-Linear: A Mixture of Experts Time Series Forecasting Model |
| |
|
| | SuperLinear is a novel time series forecasting model that employs a Mixture of Experts (MoE) architecture to achieve superior performance across various forecasting tasks. The model routes inputs to the most relevant experts based on frequency-domain analysis using FFT-based gating networks. |
| |
|
| | ## Model Architecture |
| |
|
| | The SuperLinear model consists of: |
| |
|
| | - **Sparse Mixture of Experts (MoE)**: Routes inputs to the top-k most relevant experts |
| | - **FFT-based Gating Network**: Uses frequency domain analysis to determine expert routing |
| | - **Frequency-specific Experts**: Pre-trained experts specialized for different temporal patterns |
| |
|
| | ## Key Features |
| |
|
| | - **Adaptive Expert Selection**: Dynamic routing based on input characteristics |
| | - **Frequency-aware Processing**: Leverages FFT analysis for intelligent expert selection |
| | - **Auto-regressive Capabilities**: Supports long-horizon forecasting |
| | - **Multi-scale Processing**: Handles various sequence lengths through resampling |
| |
|
| | ## Usage |
| |
|
| | ```python |
| | from transformers import AutoModelForCausalLM, AutoConfig |
| | import torch |
| | |
| | # Load the model |
| | model = AutoModelForCausalLM.from_pretrained("SequentialLearning/SuperLinear", trust_remote_code=True) |
| | |
| | # Prepare input time series data |
| | # Shape: [batch_size, channel, sequence_length] or [batch_size, sequence_length] |
| | input_data = torch.randn(1, 1, 512) |
| | |
| | # Generate predictions |
| | with torch.no_grad(): |
| | outputs = model(inputs_embeds=input_data, pred_len=96, get_prob = True) |
| | preds = outputs.logits # Predicted values |
| | probs = outputs.attentions # Expert probabilities stored here |
| | |
| | ``` |
| |
|
| | ## Configuration |
| |
|
| | Key parameters: |
| |
|
| | - `train_seq_len`: Training sequence length (default: 512) |
| | - `train_pred_len`: Training prediction length (default: 96) |
| | - `top_k_experts`: Number of experts to use (default: 12) |
| | - `use_fft`: Whether to use FFT-based gating (default: True) |
| | - `freq_experts`: Frequency-specific expert configuration |
| | - `moe_temp`: Temperature for expert selection during inference (default: 1) |
| |
|
| | ## Links |
| |
|
| | - **GitHub Repository**: [https://github.com/azencot-group/SuperLinear](https://github.com/azencot-group/SuperLinear) |
| | - **Paper**: [https://arxiv.org/abs/2509.15105](https://arxiv.org/abs/2509.15105) |
| |
|
| | ## Citation |
| |
|
| | If you use SuperLinear in your research, please cite: |
| |
|
| | ```bibtex |
| | @article{nochumsohn2025super, |
| | title={Super-Linear: A Lightweight Pretrained Mixture of Linear Experts for Time Series Forecasting}, |
| | author={Nochumsohn, Liran and Marshanski, Raz and Zisling, Hedi and Azencot, Omri}, |
| | journal={arXiv preprint arXiv:2509.15105}, |
| | year={2025} |
| | } |
| | ``` |
| |
|
| | ## License |
| |
|
| | This model is released under the MIT License. |
| |
|
