| # TSQA Adapter for ChatTime |
|
|
| This folder adapts ChatTime to the **Time-MQA TSQA** multi-task time-series QA |
| benchmark (`/mnt/share01/sqk/datasets/Time-MQA_TSQA/tmp`). It is the TSQA analog |
| of `rats40k_adapter` and is meant to fill in the **ChatTime TSQA baseline**. |
|
|
| The dataset has four source groups with free-text answers: |
| `anomaly_detection`, `classification`, `forecasting`, `open_ended` (the |
| `open_ended` group mixes `multiple_choice`, `true_false` and open questions). |
|
|
| ## How ChatTime is fed |
|
|
| ChatTime has no chat template, so prompts use ChatTime's native analysis |
| template (`utils.prompt.getPrompt`): |
|
|
| * `### Instruction` = the TSQA `question`, kept **verbatim** (including any |
| numbers embedded in the text, so forecasting keeps its absolute scale). |
| * `### Input` = the `time_series` field discretised + serialised into ChatTime's |
| native time-series tokens (`###value###`). |
| * `### Response` = the free-text `answer` (the SFT target). |
|
|
| Some context-enhanced forecasting questions are much longer than ChatTime's |
| 4096-token context (e.g. an earnings-call transcript with a tiny series). |
| Prompts are therefore **left-truncated** — keeping the serialized series, the |
| trailing ask and (in training) the full response — instead of failing. |
|
|
| ## Metrics |
|
|
| Evaluation reuses the **canonical Time-MQA evaluator** |
| (`MQA/data_utils.compute_group_metrics`), so the numbers are directly comparable |
| to the other TSQA baselines (MQA / ITFormer). Per group: |
|
|
| * `anomaly_detection`: accuracy (normal vs anomaly label parsing) |
| * `classification`: accuracy (class-label parsing) |
| * `forecasting`: MSE / MAE over parsed forecast values |
| * `open_ended`: accuracy (multiple-choice / true-false / semantic / numeric) |
|
|
| `eval_tsqa.py` writes `predictions.jsonl` and `metrics.json` (with a `by_group` |
| block) under the eval output directory. |
|
|
| ## Required inputs |
|
|
| - `MODEL_PATH`: local ChatTime model dir. Default |
| `/mnt/share01/sqk/models/ChatTime-1-7B-Chat`. |
| - `PYTHON_BIN`: Python executable. Defaults to |
| `/dev/shm/suiqk/conda_envs/scalerag-ts-v4/bin/python` (override if your env |
| lives elsewhere, e.g. `/home/suiqk/anaconda3/envs/scalerag-ts-v4/bin/python`). |
| - `MQA_DIR`: defaults to `/mnt/share01/sqk/MQA` (provides the shared evaluator). |
| - `DATA_ROOT`: defaults to `/mnt/share01/sqk/datasets/Time-MQA_TSQA/tmp`. |
|
|
| The defaults target the **4× V100** machine: FP16 LoRA with `LOAD_IN_4BIT=0`, |
| `PER_DEVICE_TRAIN_BATCH_SIZE=1`, `GRADIENT_ACCUMULATION_STEPS=16`, the 4-GPU |
| accelerate config at `/mnt/share01/sqk/ITFormer/accelerate_config.yaml`. |
| Generation defaults: `MAX_NEW_TOKENS=256`, `MAX_INPUT_TOKENS=3840`, |
| `MAX_SEQ_LENGTH=4096`. |
|
|
| ## SFT + Eval |
|
|
| ```bash |
| cd /mnt/share01/sqk/ChatTime |
| MODEL_PATH=/mnt/share01/sqk/models/ChatTime-1-7B-Chat \ |
| bash tsqa_adapter/run_sft_4gpu.sh |
| ``` |
|
|
| ## Smoke test |
|
|
| ```bash |
| cd /mnt/share01/sqk/ChatTime |
| MAX_TRAIN_SAMPLES=128 MAX_EVAL_SAMPLES=64 bash tsqa_adapter/run_sft_4gpu.sh |
| ``` |
|
|
| Outputs land in `tsqa_adapter/outputs/sft_<RUN_ID>/{adapter,eval}` and logs in |
| `tsqa_adapter/logs/`. |
|
|
