TSQA Adapter for ChatTime

This folder adapts ChatTime to the Time-MQA TSQA multi-task time-series QA benchmark (/mnt/share01/sqk/datasets/Time-MQA_TSQA/tmp). It is the TSQA analog of rats40k_adapter and is meant to fill in the ChatTime TSQA baseline.

The dataset has four source groups with free-text answers: anomaly_detection, classification, forecasting, open_ended (the open_ended group mixes multiple_choice, true_false and open questions).

How ChatTime is fed

ChatTime has no chat template, so prompts use ChatTime's native analysis template (utils.prompt.getPrompt):

### Instruction = the TSQA question, kept verbatim (including any numbers embedded in the text, so forecasting keeps its absolute scale).
### Input = the time_series field discretised + serialised into ChatTime's native time-series tokens (###value###).
### Response = the free-text answer (the SFT target).

Some context-enhanced forecasting questions are much longer than ChatTime's 4096-token context (e.g. an earnings-call transcript with a tiny series). Prompts are therefore left-truncated — keeping the serialized series, the trailing ask and (in training) the full response — instead of failing.

Metrics

Evaluation reuses the canonical Time-MQA evaluator (MQA/data_utils.compute_group_metrics), so the numbers are directly comparable to the other TSQA baselines (MQA / ITFormer). Per group:

anomaly_detection: accuracy (normal vs anomaly label parsing)
classification: accuracy (class-label parsing)
forecasting: MSE / MAE over parsed forecast values
open_ended: accuracy (multiple-choice / true-false / semantic / numeric)

eval_tsqa.py writes predictions.jsonl and metrics.json (with a by_group block) under the eval output directory.

Required inputs

MODEL_PATH: local ChatTime model dir. Default /mnt/share01/sqk/models/ChatTime-1-7B-Chat.
PYTHON_BIN: Python executable. Defaults to /dev/shm/suiqk/conda_envs/scalerag-ts-v4/bin/python (override if your env lives elsewhere, e.g. /home/suiqk/anaconda3/envs/scalerag-ts-v4/bin/python).
MQA_DIR: defaults to /mnt/share01/sqk/MQA (provides the shared evaluator).
DATA_ROOT: defaults to /mnt/share01/sqk/datasets/Time-MQA_TSQA/tmp.

The defaults target the 4× V100 machine: FP16 LoRA with LOAD_IN_4BIT=0, PER_DEVICE_TRAIN_BATCH_SIZE=1, GRADIENT_ACCUMULATION_STEPS=16, the 4-GPU accelerate config at /mnt/share01/sqk/ITFormer/accelerate_config.yaml. Generation defaults: MAX_NEW_TOKENS=256, MAX_INPUT_TOKENS=3840, MAX_SEQ_LENGTH=4096.

SFT + Eval

cd /mnt/share01/sqk/ChatTime
MODEL_PATH=/mnt/share01/sqk/models/ChatTime-1-7B-Chat \
bash tsqa_adapter/run_sft_4gpu.sh

Smoke test

cd /mnt/share01/sqk/ChatTime
MAX_TRAIN_SAMPLES=128 MAX_EVAL_SAMPLES=64 bash tsqa_adapter/run_sft_4gpu.sh

Outputs land in tsqa_adapter/outputs/sft_<RUN_ID>/{adapter,eval} and logs in tsqa_adapter/logs/.