| # AirTrackLM: LLM4STP Adapted for ADS-B Air Track Prediction |
|
|
| ## Complete Architecture & Implementation Plan |
|
|
| --- |
|
|
| ## 1. Executive Summary |
|
|
| We adapt the LLM4STP multi-feature fusion architecture (originally for maritime AIS ship trajectory prediction) to work with **ADS-B air track data**. The model uses a **decoder-only transformer** with four specialized embedding types — Prompt, Uncertainty, Geohash, and Temporal — fused together for **next-state prediction** pretraining. Once pretrained, the model is adaptable to downstream tasks like activity classification. |
|
|
| This design is grounded in published results from: |
| - **FTP-LLM** (arXiv:2501.17459) — LLaMA-3.1-8B for flight trajectory prediction |
| - **H3-CLM** (arXiv:2405.09596) — H3 geohash + causal LM for maritime trajectories |
| - **GeoFormer** (arXiv:2311.05092) — GPT-style geospatial tokenization |
| - **TrAISFormer** (arXiv:2109.03958) — Discrete tokenization of AIS features |
|
|
| --- |
|
|
| ## 2. System Architecture Overview |
|
|
| ``` |
| ┌─────────────────────────────────────────────────────────────────────┐ |
| │ RAW ADS-B INPUT │ |
| │ (timestamp, latitude, longitude, altitude) │ |
| └─────────────────────────┬───────────────────────────────────────────┘ |
| │ |
| ▼ |
| ┌─────────────────────────────────────────────────────────────────────┐ |
| │ FEATURE DERIVATION PIPELINE │ |
| │ │ |
| │ Raw: lat, lon, alt │ |
| │ Derived: COG, SOG, ROT, altitude_rate │ |
| │ Meta: timestamp → (hour, day_of_week, month) │ |
| │ │ |
| │ Output per timestep: │ |
| │ state_t = [lat, lon, alt, COG, SOG, ROT, alt_rate] │ |
| └─────────────────────────┬───────────────────────────────────────────┘ |
| │ |
| ▼ |
| ┌─────────────────────────────────────────────────────────────────────┐ |
| │ TOKENIZATION / ENCODING │ |
| │ │ |
| │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ |
| │ │ Geohash │ │ Continuous │ │ Temporal │ │ |
| │ │ Tokenizer │ │ Discretizer │ │ Encoder │ │ |
| │ │ │ │ │ │ │ │ |
| │ │ lat,lon,alt │ │ COG,SOG,ROT │ │ hour,dow, │ │ |
| │ │ → H3 cell + │ │ alt_rate │ │ month │ │ |
| │ │ alt_band │ │ → bin IDs │ │ → time IDs │ │ |
| │ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ │ |
| │ │ │ │ │ |
| │ ▼ ▼ ▼ │ |
| │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ |
| │ │ Geohash │ │ Feature │ │ Temporal │ │ |
| │ │ Embedding │ │ Embeddings │ │ Embedding │ │ |
| │ │ Table │ │ Tables │ │ Table │ │ |
| │ │ (d_model) │ │ (d_model) │ │ (d_model) │ │ |
| │ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ │ |
| │ │ │ │ │ |
| └──────────┼─────────────────┼─────────────────┼──────────────────────┘ |
| │ │ │ |
| ▼ ▼ ▼ |
| ┌─────────────────────────────────────────────────────────────────────┐ |
| │ EMBEDDING FUSION LAYER │ |
| │ │ |
| │ ┌────────────┐ ┌────────────┐ ┌────────────┐ ┌──────────────┐ │ |
| │ │ Geohash │ │ Feature │ │ Temporal │ │ Uncertainty │ │ |
| │ │ Embed │ │ Embed │ │ Embed │ │ Embed │ │ |
| │ │ (d_model) │ │ (d_model) │ │ (d_model) │ │ (d_model) │ │ |
| │ └─────┬──────┘ └─────┬──────┘ └─────┬──────┘ └──────┬───────┘ │ |
| │ │ │ │ │ │ |
| │ └──────────┬───┴──────┬───────┘ │ │ |
| │ │ │ │ │ |
| │ ▼ ▼ ▼ │ |
| │ E_state = E_geo + E_feat + E_temp + E_uncert │ |
| │ │ │ |
| │ ▼ │ |
| │ ┌───────────────────────────────────────────┐ │ |
| │ │ Prompt Embedding (prepended prefix) │ │ |
| │ │ [PROMPT_1, PROMPT_2, ..., PROMPT_k] │ │ |
| │ └───────────────────┬───────────────────────┘ │ |
| │ │ │ |
| │ ▼ │ |
| │ Input: [PROMPT_TOKENS | STATE_1 | STATE_2 | ... | STATE_T] │ |
| │ │ │ |
| │ ▼ │ |
| │ Linear Projection → d_model │ |
| │ │ │ |
| │ ▼ │ |
| │ + Positional Encoding (sinusoidal) │ |
| │ │ |
| └───────────────────────┬─────────────────────────────────────────────┘ |
| │ |
| ▼ |
| ┌─────────────────────────────────────────────────────────────────────┐ |
| │ DECODER-ONLY TRANSFORMER BACKBONE │ |
| │ │ |
| │ ┌─────────────────────────────────────────────────────┐ │ |
| │ │ Transformer Block ×N_layers │ │ |
| │ │ │ │ |
| │ │ ┌─────────────────────────────────────────┐ │ │ |
| │ │ │ Causal Multi-Head Self-Attention │ │ │ |
| │ │ │ (masked: each position attends only │ │ │ |
| │ │ │ to itself and earlier positions) │ │ │ |
| │ │ └──────────────────┬──────────────────────┘ │ │ |
| │ │ │ │ │ |
| │ │ ▼ │ │ |
| │ │ ┌─────────────────────────────────────────┐ │ │ |
| │ │ │ LayerNorm + Residual Connection │ │ │ |
| │ │ └──────────────────┬──────────────────────┘ │ │ |
| │ │ │ │ │ |
| │ │ ▼ │ │ |
| │ │ ┌─────────────────────────────────────────┐ │ │ |
| │ │ │ Feed-Forward Network │ │ │ |
| │ │ │ (Linear → GELU → Linear) │ │ │ |
| │ │ │ d_model → 4*d_model → d_model │ │ │ |
| │ │ └──────────────────┬──────────────────────┘ │ │ |
| │ │ │ │ │ |
| │ │ ▼ │ │ |
| │ │ ┌─────────────────────────────────────────┐ │ │ |
| │ │ │ LayerNorm + Residual Connection │ │ │ |
| │ │ └─────────────────────────────────────────┘ │ │ |
| │ │ │ │ |
| │ └─────────────────────────────────────────────┘ │ │ |
| │ │ |
| └───────────────────────┬─────────────────────────────────────────────┘ |
| │ |
| ▼ |
| ┌─────────────────────────────────────────────────────────────────────┐ |
| │ OUTPUT HEADS │ |
| │ │ |
| │ ┌─────────────────────────────────────────────────────────┐ │ |
| │ │ PRETRAINING: Next-State Prediction Head │ │ |
| │ │ │ │ |
| │ │ For each position t, predict state at t+1: │ │ |
| │ │ │ │ |
| │ │ h_t → Linear → softmax → P(geohash_token_{t+1}) │ │ |
| │ │ h_t → Linear → softmax → P(COG_bin_{t+1}) │ │ |
| │ │ h_t → Linear → softmax → P(SOG_bin_{t+1}) │ │ |
| │ │ h_t → Linear → softmax → P(ROT_bin_{t+1}) │ │ |
| │ │ h_t → Linear → softmax → P(alt_rate_bin_{t+1}) │ │ |
| │ │ h_t → Linear → softmax → P(alt_band_{t+1}) │ │ |
| │ │ │ │ |
| │ │ Loss = Σ CrossEntropy(predicted_feature, true_feature) │ │ |
| │ └─────────────────────────────────────────────────────────┘ │ |
| │ │ |
| │ ┌─────────────────────────────────────────────────────────┐ │ |
| │ │ DOWNSTREAM: Activity Classification Head │ │ |
| │ │ (attached after pretraining, frozen or fine-tuned) │ │ |
| │ │ │ │ |
| │ │ h_[BOS] or mean(h_1:T) → MLP → softmax → class label │ │ |
| │ └─────────────────────────────────────────────────────────┘ │ |
| │ │ |
| └─────────────────────────────────────────────────────────────────────┘ |
| ``` |
|
|
| --- |
|
|
| ## 3. The Four Embedding Types (Detailed) |
|
|
| ### 3.1 Geohash Embeddings — Spatial Position Encoding |
|
|
| **Purpose**: Encode the aircraft's 3D geographic position as a discrete token. |
|
|
| **Method**: We use **H3 hexagonal hierarchical spatial index** (Uber's H3) at resolution 5 (hex area ≈ 252 km², edge ≈ 9.85 km) for en-route flight, with an option to use resolution 7 (≈ 5.16 km², edge ≈ 1.22 km) for terminal areas. This follows the H3-CLM paper's approach but adapted for aviation's larger spatial scale. |
|
|
| **3D Extension**: Since aircraft operate in 3D, we combine the H3 cell with an **altitude band**: |
| ``` |
| Geohash Token = H3_cell_index × N_alt_bands + alt_band_index |
| |
| Altitude bands (1000 ft increments): |
| Band 0: 0 - 1,000 ft (ground / taxi) |
| Band 1: 1,000 - 2,000 ft (initial climb / approach) |
| ... |
| Band 45: 44,000 - 45,000 ft (high cruise) |
| |
| N_alt_bands = 46 |
| ``` |
|
|
| **Vocabulary size**: At H3 resolution 5, the number of unique cells covering typical airspace is ~100K-200K. With altitude bands: `~200K × 46 ≈ 9.2M` — too large for direct embedding. |
|
|
| **Solution — Factored Embedding**: |
| ``` |
| E_geohash = E_h3[h3_cell_id] + E_alt[alt_band_id] |
| |
| E_h3: learned embedding table, vocab = N_h3_cells (~200K or hashing trick to 50K) |
| E_alt: learned embedding table, vocab = 46 |
| |
| Both project to d_model dimensions. |
| ``` |
|
|
| The **hashing trick**: Map H3 cell indices through a hash function to a fixed vocabulary of ~50,000 buckets. This bounds memory while maintaining spatial discrimination. |
|
|
| **Why H3 over traditional geohash**: H3 hexagons have uniform area (no polar distortion), hierarchical nesting, and consistent neighbor relationships — critical for trajectory continuity. |
|
|
| ### 3.2 Temporal Embeddings — When Is the Aircraft Flying? |
|
|
| **Purpose**: Encode temporal context — time of day affects traffic density, routes, and behavior. |
|
|
| **Method**: Additive composition of multiple temporal scales: |
| ``` |
| E_temporal = E_hour[hour_of_day] + E_dow[day_of_week] + E_month[month] |
| |
| E_hour: 24 entries (captures rush hour vs. night patterns) |
| E_dow: 7 entries (weekday vs. weekend traffic) |
| E_month: 12 entries (seasonal routes, weather patterns) |
| |
| All project to d_model dimensions. |
| ``` |
|
|
| **Optional — Sinusoidal Sub-minute Encoding**: For sub-minute resolution: |
| ``` |
| E_minute = sin(2π × minute / 60), cos(2π × minute / 60) → linear → d_model |
| ``` |
|
|
| ### 3.3 Uncertainty Embeddings — How Confident Are We? |
|
|
| **Purpose**: Encode the model's uncertainty about the current trajectory state. Aircraft in straight-and-level cruise have low uncertainty; aircraft maneuvering near airports have high uncertainty. |
|
|
| **Method**: Compute a **trajectory smoothness score** from recent states, then discretize: |
|
|
| ``` |
| Uncertainty sources (sliding window of k=5 recent states): |
| |
| 1. Position variance: σ²_pos = var(Δlat) + var(Δlon) |
| 2. Heading variance: σ²_COG = circular_var(COG_{t-k:t}) |
| 3. Speed variance: σ²_SOG = var(SOG_{t-k:t}) |
| 4. Altitude variance: σ²_alt = var(alt_rate_{t-k:t}) |
| |
| Combined uncertainty score: |
| U_t = w1·σ²_pos + w2·σ²_COG + w3·σ²_SOG + w4·σ²_alt |
| |
| Discretize into N_uncert = 16 bins (quantile binning on training data) |
| |
| E_uncertainty = E_uncert_table[bin(U_t)] → d_model |
| ``` |
|
|
| **Weights w1-w4**: Hyperparameters tuned on validation data, or learned as part of the model. |
|
|
| **During inference**: For multi-step prediction, uncertainty can be updated using MC-Dropout or ensemble disagreement. |
|
|
| ### 3.4 Prompt Embeddings — Task and Context Metadata |
|
|
| **Purpose**: Provide metadata context about the flight, analogous to system prompts in LLMs. Enables task conditioning and multi-task learning. |
|
|
| **Method**: Learnable prompt tokens prepended to the trajectory: |
|
|
| ``` |
| Prompt token vocabulary: |
| - Aircraft category: [HEAVY, LARGE, SMALL, ROTORCRAFT, GLIDER, UAV, UNKNOWN] (7) |
| - Flight phase: [CLIMB, CRUISE, DESCENT, APPROACH, GROUND, UNKNOWN] (6) |
| - Region: [CONUS, EUROPE, ASIA, OTHER] (4) |
| - Task: [PREDICT, CLASSIFY, DETECT_ANOMALY] (3) |
| - Special: [BOS, EOS, PAD, MASK] (4) |
| |
| Total prompt vocab: ~24 tokens |
| |
| Prompt sequence (prepended): |
| [BOS, TASK_TOKEN, AIRCRAFT_TOKEN, PHASE_TOKEN, REGION_TOKEN] |
| |
| Each has a learned embedding of dimension d_model. |
| ``` |
|
|
| **For downstream classification**: Change TASK_TOKEN to CLASSIFY; output at BOS position is used for classification. |
| |
| --- |
| |
| ## 4. Feature Derivation Pipeline |
| |
| ### 4.1 Raw Input |
| ``` |
| timestamp (Unix epoch seconds) |
| latitude (degrees, WGS84) |
| longitude (degrees, WGS84) |
| altitude (feet, barometric or geometric) |
| ``` |
| |
| ### 4.2 Derived Features |
| |
| ```python |
| import numpy as np |
| |
| def derive_features(timestamps, lats, lons, alts): |
| """ |
| Derive COG, SOG, ROT, and altitude rate from raw position data. |
| All inputs: numpy arrays of shape (N,) for a single trajectory. |
| Returns arrays of shape (N,) — first element is NaN. |
| """ |
| dt = np.diff(timestamps) # seconds |
| dt = np.maximum(dt, 1e-6) # avoid division by zero |
| |
| # --- Course Over Ground (COG) --- |
| lat1, lat2 = np.radians(lats[:-1]), np.radians(lats[1:]) |
| dlon = np.radians(np.diff(lons)) |
| |
| x = np.sin(dlon) * np.cos(lat2) |
| y = np.cos(lat1) * np.sin(lat2) - np.sin(lat1) * np.cos(lat2) * np.cos(dlon) |
| COG = np.degrees(np.arctan2(x, y)) % 360 # [0, 360) |
| |
| # --- Speed Over Ground (SOG) --- |
| dlat = np.radians(np.diff(lats)) |
| a = np.sin(dlat/2)**2 + np.cos(lat1) * np.cos(lat2) * np.sin(dlon/2)**2 |
| c = 2 * np.arctan2(np.sqrt(a), np.sqrt(1-a)) |
| distance_nm = 3440.065 * c # Earth radius in nautical miles |
| SOG = distance_nm / (dt / 3600) # knots |
| |
| # --- Rate of Turn (ROT) --- |
| dCOG = np.diff(COG) |
| dCOG = (dCOG + 180) % 360 - 180 # normalize to [-180, 180] |
| ROT = np.full(len(lats), np.nan) |
| ROT[2:] = dCOG / dt[1:] # degrees per second |
| |
| # --- Rate of Altitude Change --- |
| dalt = np.diff(alts) # feet |
| alt_rate = dalt / (dt / 60) # feet per minute |
| |
| # Pad first elements |
| COG_full = np.concatenate([[np.nan], COG]) |
| SOG_full = np.concatenate([[np.nan], SOG]) |
| alt_rate_full = np.concatenate([[np.nan], alt_rate]) |
| |
| return COG_full, SOG_full, ROT, alt_rate_full |
| ``` |
| |
| ### 4.3 Feature Discretization |
|
|
| | Feature | Range | Bin Width | N_bins | Notes | |
| |---------------|-------------------|--------------|--------|--------------------| |
| | COG | [0, 360) | 5° | 72 | Circular | |
| | SOG | [0, 600] kts | 5 knots | 121 | Capped at ~Mach 1 | |
| | ROT | [-6, 6] °/s | 0.25 °/s | 49 | Capped ±6°/s | |
| | Altitude Rate | [-6000, 6000] fpm | 200 ft/min | 61 | Capped ±6000 fpm | |
| |
| Outliers beyond caps clipped to boundary bin. |
| |
| ### 4.4 Trajectory Preprocessing Pipeline |
| |
| ``` |
| 1. Segment raw ADS-B by ICAO24 + temporal gaps > 15 min → individual flights |
| 2. Resample to fixed Δt = 60 seconds (linear interp for position, circular for heading) |
| 3. Derive features (COG, SOG, ROT, alt_rate) |
| 4. Drop first 2 points per trajectory (NaN from derivation) |
| 5. Filter: remove trajectories with < 20 points (< 20 minutes) |
| 6. Compute H3 cell (res 5) + altitude band for each point |
| 7. Discretize all continuous features into bins |
| 8. Compute uncertainty scores (sliding window k=5) |
| 9. Extract temporal features (hour, dow, month) |
| 10. Construct prompt tokens from metadata (if available) |
| ``` |
| |
| --- |
| |
| ## 5. Model Hyperparameters |
| |
| ### 5.1 Model Dimensions |
| |
| | Parameter | Value | Rationale | |
| |------------------|--------|----------------------------------------------------| |
| | d_model | 256 | H3-CLM found 256-1024 effective | |
| | n_heads | 8 | head_dim = 32 | |
| | n_layers | 8 | Moderate depth for ~10M param model | |
| | d_ff | 1024 | 4× d_model (standard) | |
| | max_seq_len | 128 | 128 states × 60s ≈ 2 hours of flight | |
| | n_prompt_tokens | 5 | [BOS, TASK, AIRCRAFT, PHASE, REGION] | |
| | dropout | 0.1 | | |
| |
| **Total parameters**: ~8-12M (trainable on single GPU in hours) |
| |
| ### 5.2 Vocabulary Sizes |
| |
| | Embedding | Vocab | Dim | |
| |------------------|--------|-----| |
| | H3 cells | 50,000 | 256 | |
| | Altitude bands | 46 | 256 | |
| | COG bins | 72 | 256 | |
| | SOG bins | 121 | 256 | |
| | ROT bins | 49 | 256 | |
| | Alt rate bins | 61 | 256 | |
| | Hour of day | 24 | 256 | |
| | Day of week | 7 | 256 | |
| | Month | 12 | 256 | |
| | Uncertainty bins | 16 | 256 | |
| | Prompt tokens | 24 | 256 | |
| |
| ### 5.3 State Token Composition |
| |
| Each timestep → single state token via additive fusion: |
| |
| ``` |
| E_state_t = E_h3[h3_id_t] + E_alt_band[alt_band_t] # Geohash (3D position) |
| + E_COG[cog_bin_t] + E_SOG[sog_bin_t] # Kinematics |
| + E_ROT[rot_bin_t] + E_alt_rate[alt_rate_bin_t] # Dynamics |
| + E_hour[hour_t] + E_dow[dow_t] + E_month[month_t] # Temporal |
| + E_uncert[uncert_bin_t] # Uncertainty |
|
|
| E_state_t ∈ R^{d_model} |
| ``` |
| |
| This additive fusion follows BERT (token + segment + position) and TrAISFormer. |
| |
| --- |
| |
| ## 6. Training Recipe |
| |
| ### 6.1 Pretraining: Next-State Prediction (Causal LM) |
| |
| **Objective**: Given states 1..T, predict state at T+1 (applied autoregressively at every position). |
| |
| **Loss**: |
| ``` |
| L = Σ_{t=1}^{T-1} [ λ_geo · CE(ŷ_geo_t, y_geo_{t+1}) |
| + λ_COG · CE(ŷ_COG_t, y_COG_{t+1}) |
| + λ_SOG · CE(ŷ_SOG_t, y_SOG_{t+1}) |
| + λ_ROT · CE(ŷ_ROT_t, y_ROT_{t+1}) |
| + λ_alt · CE(ŷ_alt_rate_t, y_alt_rate_{t+1}) |
| + λ_altb · CE(ŷ_alt_band_t, y_alt_band_{t+1}) ] |
|
|
| λ values default to 1.0 (equal weighting). |
| ``` |
| |
| **Training hyperparameters** (based on FTP-LLM + H3-CLM): |
| |
| | Parameter | Value | |
| |----------------------|---------------------| |
| | Optimizer | AdamW | |
| | Learning rate | 5e-4 | |
| | LR Schedule | Cosine + 5% warmup | |
| | Batch size (per GPU) | 64 | |
| | Gradient accumulation| 4 (effective = 256) | |
| | Max epochs | 30 (early stop p=5) | |
| | Weight decay | 0.01 | |
| | Gradient clipping | 1.0 | |
| | Mixed precision | bf16 | |
| |
| **Data windowing**: Sliding window size=128, stride=64 (50% overlap). |
| |
| ### 6.2 Downstream: Activity Classification |
| |
| After pretraining, attach classification head: |
| ``` |
| h_BOS → Linear(256, 128) → GELU → Dropout(0.1) → Linear(128, N_classes) |
| ``` |
| |
| **Fine-tuning options**: |
| - **A**: Freeze backbone, train head only (fast, small data) |
| - **B**: Full fine-tune, backbone lr=1e-5, head lr=1e-3 |
| |
| --- |
| |
| ## 7. Dataset Strategy |
| |
| ### 7.1 Prototyping — `traffic` Python Library |
| |
| ```python |
| from traffic.data.samples import landing_zurich_2019 |
| # ~2,000 flights near Zurich |
| # Columns: timestamp, icao24, callsign, latitude, longitude, altitude, |
| # groundspeed, track, vertical_rate, ... |
| ``` |
| |
| Instant access, clean, well-documented. Single airport, limited diversity. |
| |
| ### 7.2 Training — OpenSky Network |
| |
| ```python |
| from pyopensky.trino import Trino |
| trino = Trino() |
| df = trino.rawquery(""" |
| SELECT time, icao24, lat, lon, baroaltitude, velocity, heading, vertrate |
| FROM state_vectors_data4 |
| WHERE hour >= '2024-01-15 00:00:00' |
| AND hour < '2024-01-15 12:00:00' |
| AND lat BETWEEN 40 AND 55 |
| AND lon BETWEEN -10 AND 20 |
| ORDER BY icao24, time |
| """) |
| ``` |
| |
| **Target**: |
| - **Region A** (train): Europe, 1 month → ~500K-1M flights |
| - **Region B** (OOD test): US CONUS, 1 week → ~200K flights |
| - **Region C** (far test): East Asia, 1 week → ~100K flights |
| |
| ### 7.3 Alternative: SCAT Dataset |
| |
| ~170K en-route flights over Sweden, Zenodo. Pre-segmented, clean. |
| |
| ### 7.4 Data Split |
| |
| ``` |
| Training: 70% of Region A flights |
| Validation: 15% of Region A flights |
| Test (IID): 15% of Region A flights |
| Test (OOD): 100% of Region B flights |
| Test (Far): 100% of Region C flights |
| ``` |
| |
| Split by **flight** (not time window) to avoid data leakage. |
| |
| --- |
| |
| ## 8. Ablation Study: Geohash Geographic Dependency |
| |
| ### 8.1 Hypothesis |
| |
| > Geohash embeddings encode **absolute geographic position**, causing the model to memorize region-specific patterns (airways, approach paths, airspace structure). This improves in-distribution performance but degrades transfer to unseen regions. |
| |
| ### 8.2 Experimental Variants |
| |
| | Variant | Geohash Type | Description | |
| |---------|-------------|-------------| |
| | **V1: Full Model** | H3 absolute | Complete architecture as described | |
| | **V2: No Geohash** | None | Remove geohash entirely; model sees only kinematics + temporal + uncertainty | |
| | **V3: Relative Geohash** | H3 relative | H3 cell of (Δlat, Δlon) from trajectory start — position-invariant | |
| | **V4: Multi-Resolution** | H3 res 3+5+7 | 3 resolutions summed (coarse→fine) | |
| | **V5: Continuous Position** | Linear projection | `Linear([lat, lon, alt] → d_model)` — no discretization | |
|
|
| ### 8.3 Evaluation Metrics |
|
|
| For each variant × each test set (IID, OOD, Far): |
|
|
| | Metric | Description | |
| |--------|-------------| |
| | Geo Accuracy | % correct H3 cell prediction | |
| | Position MAE | Mean absolute error in km | |
| | COG MAE | Heading error in degrees | |
| | SOG MAE | Speed error in knots | |
| | Multi-step ADE | Average displacement error over 5 predicted steps | |
| | Multi-step FDE | Final displacement error at step 5 | |
|
|
| ### 8.4 Key Comparisons |
|
|
| | Comparison | Tests | |
| |-----------|-------| |
| | V1 vs V2 (IID) | How much geohash helps when test = train region | |
| | V1 vs V2 (OOD) | If V2 > V1 on OOD → geohash causes geographic overfitting | |
| | V1 vs V3 (OOD) | If V3 good on both IID and OOD → relative geohash is the sweet spot | |
| | V4 (all) | Multi-resolution: coarse cells transfer, fine cells specialize? | |
| | V5 (all) | Does continuous encoding avoid discretization issues? | |
|
|
| ### 8.5 Expected Outcomes |
|
|
| - **V1**: Best IID, worst OOD (hypothesis) |
| - **V3**: Best compromise — predicted winner |
| - **V5**: May struggle (loses discrete token structure transformers excel at) |
| - **V2**: Strong OOD baseline, sacrifices IID |
|
|
| ### 8.6 Additional Analysis |
|
|
| - **Attention visualization**: V1 vs V3 attention patterns |
| - **Embedding clustering**: t-SNE of geohash embeddings colored by region |
| - **Learning curves**: IID vs OOD performance vs training data size |
|
|
| --- |
|
|
| ## 9. Implementation Phases |
|
|
| ### Phase 1: Data Pipeline (Week 1) |
| - Set up `traffic` library, extract sample trajectories |
| - Implement feature derivation (COG, SOG, ROT, alt_rate) |
| - Implement H3 geohash encoding + altitude banding |
| - Implement feature discretization (binning) |
| - Implement uncertainty score computation |
| - Build PyTorch Dataset class with sliding window |
| - Unit tests for all derivation functions |
| |
| ### Phase 2: Model Architecture (Week 1-2) |
| - Implement all embedding tables |
| - Implement additive fusion layer |
| - Implement prompt token prepending |
| - Implement decoder-only transformer backbone |
| - Implement multi-head output (6 prediction heads) |
| - Implement classification head (for downstream) |
| - Forward pass test with dummy data |
| |
| ### Phase 3: Pretraining (Week 2-3) |
| - Implement training loop with multi-task loss |
| - Prototyping run on `traffic` data (small, fast iteration) |
| - Scale to OpenSky data |
| - Monitor loss curves, validate convergence |
| - Save best checkpoint |
| |
| ### Phase 4: Downstream Adaptation (Week 3-4) |
| - Implement classification fine-tuning pipeline |
| - Test on activity classification task |
| - Compare frozen vs. fine-tuned backbone |
| |
| ### Phase 5: Ablation Study (Week 4-5) |
| - Implement all 5 geohash variants |
| - Train each variant with identical hyperparameters |
| - Evaluate on IID, OOD, and Far test sets |
| - Generate comparison tables and visualizations |
| - Write analysis of geographic dependency findings |
| |
| --- |
| |
| ## 10. Key Design Decisions & Rationale |
| |
| | Decision | Choice | Why | |
| |----------|--------|-----| |
| | Custom model vs. pretrained LLM | Custom ~10M param transformer | FTP-LLM showed text-tokenized LLMs work, but custom allows proper multi-feature fusion. 10M params trains in hours. | |
| | H3 vs. traditional geohash | H3 | Uniform hexagonal cells, no polar distortion, hierarchical. Proven by H3-CLM. | |
| | Additive vs. concatenative fusion | Additive | BERT/TrAISFormer paradigm. Keeps d_model constant. Concatenation → d_model × N_features = massive. | |
| | 60s time resolution | 60 seconds | FTP-LLM validated 1-min aggregation. 128 steps ≈ 2+ hours. | |
| | Factored geohash (H3 + alt) | Separate tables, summed | Avoids combinatorial explosion (9.2M → 50K + 46). | |
| | Multi-head output | Separate softmax per feature | More interpretable, allows per-feature analysis. | |
| | Uncertainty from smoothness | Variance-based | Computable at data time, no inference overhead. | |
|
|
| --- |
|
|
| ## 11. Risk Analysis |
|
|
| | Risk | Likelihood | Impact | Mitigation | |
| |------|-----------|--------|------------| |
| | Geohash overfits to region | High | High | Ablation study; V3 (relative) is fallback | |
| | OpenSky access issues | Medium | High | Fallback: `traffic` samples + SCAT | |
| | 60s too coarse for terminal | Medium | Low | Separate terminal model at 10s | |
| | Model too small | Low | Medium | Scale: d_model→512, n_layers→16 (~40M) | |
| | Alt discretization too coarse | Low | Low | Refine to 500ft bands (92) | |
|
|
| --- |
|
|
| ## 12. Monitoring & Evaluation |
|
|
| **During training** (Trackio): |
| - Total loss + per-feature loss curves |
| - Validation loss each epoch |
| - LR schedule, GPU utilization |
|
|
| **After training**: |
| - Next-state accuracy (top-1, top-5 per feature) |
| - Position error in km |
| - Multi-step prediction (1, 5, 10, 20 steps ahead) |
| - Downstream classification F1/precision/recall |
|
|
| --- |
|
|
| *Grounded in: FTP-LLM, H3-CLM, GeoFormer, TrAISFormer, and LLM4STP (reconstructed). Ready for implementation upon approval.* |
|
|
