Hugging Face's logo

a12354
/
ITFormer

Safetensors
Model card Files
xet
 Community
ITFormer / .github /copilot-instructions.md
a12354's picture
a12354
Add files using upload-large-folder tool
c8aad8f verified
preview code
|
Raw
History Blame Contribute Delete
13.3 kB

ITFormer - Temporal-Textual Multimodal Question Answering Framework

ALWAYS follow these instructions first and only fallback to additional search and context gathering if the information here is incomplete or found to be in error.

Overview

ITFormer is a Python-based deep learning framework for temporal-textual multimodal question answering. It combines time series data with natural language processing using transformer architectures and large language models (Qwen2.5). The project requires GPU resources for optimal performance but can run on CPU for testing.

Working Effectively

Environment Setup and Dependencies

NEVER CANCEL builds or installs - they may take 10+ minutes. Set timeout to 15+ minutes minimum.

# Core Python dependencies - takes ~3-5 minutes. NEVER CANCEL.
pip install -r requirements.txt

# Additional required dependencies not in requirements.txt
pip install h5py nltk rouge-score scikit-learn

Expected timing:

  • Main dependency installation: 3-5 minutes
  • Additional dependencies: 1-2 minutes
  • CRITICAL: Set timeout to 15+ minutes for pip installs to avoid premature cancellation

Project Structure Requirements

The project requires a specific directory structure to function correctly:

ITFormer-ICML25/
├── dataset/
│   └── datasets/                    # EngineMT-QA dataset files
│       ├── time_series_data.h5      # Time series data (required for inference)
│       ├── train_qa.jsonl           # Training QA pairs
│       └── test_qa.jsonl            # Test QA pairs (required for inference)
├── LLM/                             # Base LLM models directory
│   ├── Qwen2.5-0.5B-Instruct/       # For ITFormer-0.5B
│   ├── Qwen2.5-3B-Instruct/         # For ITFormer-3B  
│   └── Qwen2.5-7B-Instruct/         # For ITFormer-7B
├── checkpoints/                     # ITFormer model checkpoints
│   ├── ITFormer-0.5B/               # Lightweight model (beats ChatGPT-4o)
│   ├── ITFormer-3B/                 # Medium model
│   └── ITFormer-7B/                 # Best performance model
└── inference_results/               # Generated output directory

Model and Dataset Downloads

CRITICAL NETWORK REQUIREMENT: Model downloads require internet access and may fail in restricted environments.

Model download timing - NEVER CANCEL:

  • ITFormer-0.5B: ~15-30 minutes (smallest model)
  • ITFormer-7B: ~60-120 minutes (largest model)
  • Qwen2.5 models: ~30-90 minutes each depending on size
  • EngineMT-QA dataset: ~10-20 minutes
# Create required directories
mkdir -p LLM checkpoints dataset/datasets

# Install Git LFS for large model downloads  
git lfs install

# Download models (choose based on your needs - 0.5B recommended for testing)
# Set timeout to 120+ minutes for large model downloads
python -c "from huggingface_hub import snapshot_download; snapshot_download(repo_id='pandalin98/ITFormer-0.5B', local_dir='./checkpoints/ITFormer-0.5B')"
python -c "from huggingface_hub import snapshot_download; snapshot_download(repo_id='Qwen/Qwen2.5-0.5B-Instruct', local_dir='./LLM/Qwen2.5-0.5B-Instruct')"

# Download dataset
python -c "from huggingface_hub import snapshot_download; snapshot_download(repo_id='pandalin98/EngineMT-QA', repo_type='dataset', local_dir='./dataset/datasets')"

Running Inference

Primary inference methods:

# Method 1: Direct Python execution (single GPU/CPU)
python inference.py --config yaml/infer.yaml

# Method 2: Multi-GPU with Accelerate (recommended for production)
# Takes ~5-30 minutes depending on dataset size and model. NEVER CANCEL.
accelerate launch --config_file accelerate_config.yaml inference.py --config yaml/infer.yaml

# Method 3: Using provided script
bash infra.bash

Model size options:

# ITFormer-0.5B (default) - fastest, good performance
python inference.py --config yaml/infer.yaml --model_checkpoint checkpoints/ITFormer-0.5B

# ITFormer-3B - medium performance  
python inference.py --config yaml/infer.yaml --model_checkpoint checkpoints/ITFormer-3B

# ITFormer-7B - best performance, slowest
python inference.py --config yaml/infer.yaml --model_checkpoint checkpoints/ITFormer-7B

Expected inference timing - NEVER CANCEL:

  • Small dataset (100 samples): 2-5 minutes
  • Medium dataset (1000 samples): 10-20 minutes
  • Full dataset (10k+ samples): 30-60 minutes
  • Set timeout to 90+ minutes for full inference runs

Validation and Testing

Always run these validation steps after making changes:

# 1. Test core imports (quick validation)
python -c "from models.TimeLanguageModel import TLM, TLMConfig; from dataset.dataset import TsQaDataset; from utils.metrics import open_question_metrics; print('✅ All modules import successfully')"

# 2. Test inference script help (validates argument parsing)
python inference.py --help

# 3. Syntax validation for all Python files
find . -name "*.py" -exec python -m py_compile {} \;

# 4. Test basic model initialization (requires models to be downloaded)
python -c "
import yaml
with open('yaml/infer.yaml', 'r') as f:
    config = yaml.safe_load(f)
print('✅ Configuration file loads correctly')
"

Configuration

Main configuration file: yaml/infer.yaml

Key parameters you might need to modify:

  • ts_path_test: Path to time series data file
  • qa_path_test: Path to QA pairs file
  • batch_size: Inference batch size (default: 12)
  • GPU-related settings handled by accelerate_config.yaml

Common Issues and Solutions

Import errors: Missing dependencies

# Error: ModuleNotFoundError: No module named 'h5py'
# Solution: Install additional dependencies
pip install h5py nltk rouge-score scikit-learn

"Checkpoint path does not exist": Missing model files

# Error: ValueError: Checkpoint path does not exist: checkpoints/ITFormer-0.5B
# Solution: Download required models or create directory structure
mkdir -p checkpoints/ITFormer-0.5B LLM/Qwen2.5-0.5B-Instruct dataset/datasets

Network/download failures: Restricted internet access

# Error: requests.exceptions.ConnectionError or LocalEntryNotFoundError
# Solution: Download models on a machine with internet access, then transfer
# Alternative: Use offline model loading if models are already available locally

CUDA out of memory: GPU memory insufficient

# Error: RuntimeError: CUDA out of memory
# Solution: Reduce batch size in yaml/infer.yaml
# Or run on CPU (slower but works): add --cpu flag to accelerate launch

Accelerate multi-process errors: Configuration mismatch

# Error: Process rank errors or device count mismatches
# Solution: Run with single process first, then debug multi-GPU setup
python inference.py --config yaml/infer.yaml  # Instead of accelerate launch

Missing dataset files: Data files not found

# Error: FileNotFoundError for time_series_data.h5 or test_qa.jsonl
# Solution: Ensure dataset is downloaded to correct location
ls -la dataset/datasets/  # Should show time_series_data.h5 and test_qa.jsonl

Permission/filesystem errors: Directory access issues

# Error: PermissionError or filesystem access denied
# Solution: Ensure you have write permissions to the working directory
chmod -R 755 . && mkdir -p checkpoints LLM dataset/datasets inference_results

Key Components

Core modules locations:

  • inference.py: Main inference script and entry point
  • models/TimeLanguageModel.py: Core ITFormer model implementation
  • models/TimeSeriesEncoder.py: Time series encoding components
  • dataset/dataset.py: Data loading and preprocessing
  • utils/metrics.py: Evaluation metrics for QA performance
  • yaml/infer.yaml: Main configuration file
  • accelerate_config.yaml: Multi-GPU acceleration settings

Model architecture: ITFormer combines:

  • Time series encoder (processes temporal data)
  • Large language model (Qwen2.5 variants)
  • Cross-modal attention mechanisms
  • Question-answering head for generation

Performance Expectations

ITFormer model performance:

  • 0.5B model: Beats ChatGPT-4o on temporal QA, fastest inference
  • 3B model: Balanced performance and speed
  • 7B model: Best accuracy, slowest inference

Resource requirements:

  • Minimum RAM: 8GB (for 0.5B model)
  • Recommended RAM: 16GB+ (for 3B+)
  • GPU memory: 4GB+ (0.5B), 12GB+ (7B)
  • Storage: 50GB+ for all models and dataset

Development Workflow

  1. Always validate imports first - run the import validation command
  2. Test with smallest model (0.5B) before trying larger models
  3. Use CPU mode for testing when GPU resources are limited
  4. Monitor memory usage during inference runs
  5. Check output directory inference_results/ for generated results and metrics
  6. Verify configuration paths in yaml/infer.yaml match your directory structure

Troubleshooting Build/Runtime Issues

For any "module not found" errors: Re-run the dependency installation commands above For "file not found" errors: Verify directory structure matches the required layout
For memory errors: Reduce batch_size in configuration or use smaller model For slow performance: Ensure GPU is available and properly configured with accelerate

Complete Setup Workflow for New Clones

When working with a fresh clone, follow this exact sequence:

# 1. Install core dependencies (3-5 minutes, NEVER CANCEL)
pip install -r requirements.txt

# 2. Install additional dependencies (1-2 minutes) 
pip install h5py nltk rouge-score scikit-learn

# 3. Verify imports work
python -c "from models.TimeLanguageModel import TLM, TLMConfig; from dataset.dataset import TsQaDataset; from utils.metrics import open_question_metrics; print('✅ All modules import successfully')"

# 4. Create directory structure
mkdir -p LLM checkpoints dataset/datasets

# 5. Download models and dataset (30-120+ minutes total, NEVER CANCEL)
# Only if you have internet access - skip if in restricted environment
python -c "from huggingface_hub import snapshot_download; snapshot_download(repo_id='pandalin98/ITFormer-0.5B', local_dir='./checkpoints/ITFormer-0.5B')"
python -c "from huggingface_hub import snapshot_download; snapshot_download(repo_id='Qwen/Qwen2.5-0.5B-Instruct', local_dir='./LLM/Qwen2.5-0.5B-Instruct')"
python -c "from huggingface_hub import snapshot_download; snapshot_download(repo_id='pandalin98/EngineMT-QA', repo_type='dataset', local_dir='./dataset/datasets')"

# 6. Test inference pipeline (only works if models/dataset are downloaded)
python inference.py --help  # Should show help without errors

Repository Structure Reference

Current repository contents (useful for navigation):

ITFormer-ICML25/
├── .github/
│   └── copilot-instructions.md      # This file
├── models/                          # Core model implementations  
│   ├── TimeLanguageModel.py         # Main ITFormer model
│   ├── TimeSeriesEncoder.py         # Time series processing
│   ├── TT_Former.py                 # Transformer components
│   └── layers/                      # Neural network layers
├── dataset/                         # Data handling
│   └── dataset.py                   # Dataset loading logic
├── utils/                           # Utility functions
│   ├── metrics.py                   # Evaluation metrics
│   ├── position_coding.py           # Positional encodings
│   └── gradio_vlm.py               # UI components
├── yaml/
│   └── infer.yaml                   # Main configuration file
├── inference.py                     # Main entry point script
├── requirements.txt                 # Core dependencies
├── accelerate_config.yaml          # Multi-GPU settings
├── infra.bash                      # Shortcut script
└── README.md                       # Project documentation

Key files you'll modify most often:

  • inference.py: Main inference logic and entry point
  • yaml/infer.yaml: Configuration parameters
  • models/TimeLanguageModel.py: Core model architecture
  • dataset/dataset.py: Data loading and preprocessing

Common Development Tasks

Testing changes without full model downloads:

# Basic validation that doesn't require models
python -c "from models.TimeLanguageModel import TLM, TLMConfig; print('Core models loadable')"
python inference.py --help
find . -name "*.py" -exec python -m py_compile {} \;

Monitoring inference progress:

# Watch output directory for results
watch ls -la inference_results/

# Monitor GPU usage (if available)
nvidia-smi -l 5

Quick syntax and import checks:

# Check all Python files compile
find . -name "*.py" -exec python -m py_compile {} \; && echo "✅ All Python files compile successfully"

# Test core framework imports
python -c "
import torch
from transformers import AutoTokenizer
from accelerate import Accelerator
print('✅ Core ML libraries work')
"

Remember: This is a research framework optimized for temporal-textual QA tasks. Always start with the 0.5B model for testing and validation before moving to larger models.