{
 "nbformat": 4,
 "nbformat_minor": 0,
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ant-research/EasyTemporalPointProcess/blob/main/notebooks/easytpp_3_train_eval.ipynb)\n",
    "\n",
    "# Tutorial 3：Training and Evaluation Pipeline in EasyTPP\n",
    "\n",
    "In this tutorial, we'll walk through the complete process of training and evaluating temporal point process (TPP) models using the **EasyTPP** framework.\n",
    "\n",
    "In this notebook, we will cover the following key aspects:\n",
    "- **Data Preparation**: Loading and preprocessing event sequence data.\n",
    "- **Model Training**: Configuring and training a Neural Hawkes Process (NHP) model.\n",
    "- **Model Evaluation**: Assessing model performance using various metrics.\n",
    "- **Visualization**: Analyzing and visualizing model predictions and results.\n",
    "\n",
    "We begin by installing the package"
   ],
   "metadata": {
    "id": "mprLutjnft_a"
   }
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "collapsed": true,
    "id": "aH28ufHMa-QU",
    "outputId": "2e4ca8e1-41a4-4c44-deb5-9fc0a904a2fc"
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Requirement already satisfied: easy-tpp in /usr/local/lib/python3.11/dist-packages (0.1.0)\n",
      "Collecting easy-tpp\n",
      "  Downloading easy_tpp-0.1.2-py3-none-any.whl.metadata (533 bytes)\n",
      "Requirement already satisfied: PyYAML>=5.1 in /usr/local/lib/python3.11/dist-packages (from easy-tpp) (6.0.2)\n",
      "Requirement already satisfied: numpy in /usr/local/lib/python3.11/dist-packages (from easy-tpp) (2.0.2)\n",
      "Requirement already satisfied: pandas in /usr/local/lib/python3.11/dist-packages (from easy-tpp) (2.2.2)\n",
      "Requirement already satisfied: torch in /usr/local/lib/python3.11/dist-packages (from easy-tpp) (2.6.0+cu124)\n",
      "Requirement already satisfied: tensorboard in /usr/local/lib/python3.11/dist-packages (from easy-tpp) (2.18.0)\n",
      "Requirement already satisfied: packaging in /usr/local/lib/python3.11/dist-packages (from easy-tpp) (24.2)\n",
      "Requirement already satisfied: datasets in /usr/local/lib/python3.11/dist-packages (from easy-tpp) (3.5.0)\n",
      "Requirement already satisfied: omegaconf in /usr/local/lib/python3.11/dist-packages (from easy-tpp) (2.3.0)\n",
      "Requirement already satisfied: filelock in /usr/local/lib/python3.11/dist-packages (from datasets->easy-tpp) (3.18.0)\n",
      "Requirement already satisfied: pyarrow>=15.0.0 in /usr/local/lib/python3.11/dist-packages (from datasets->easy-tpp) (18.1.0)\n",
      "Requirement already satisfied: dill<0.3.9,>=0.3.0 in /usr/local/lib/python3.11/dist-packages (from datasets->easy-tpp) (0.3.8)\n",
      "Requirement already satisfied: requests>=2.32.2 in /usr/local/lib/python3.11/dist-packages (from datasets->easy-tpp) (2.32.3)\n",
      "Requirement already satisfied: tqdm>=4.66.3 in /usr/local/lib/python3.11/dist-packages (from datasets->easy-tpp) (4.67.1)\n",
      "Requirement already satisfied: xxhash in /usr/local/lib/python3.11/dist-packages (from datasets->easy-tpp) (3.5.0)\n",
      "Requirement already satisfied: multiprocess<0.70.17 in /usr/local/lib/python3.11/dist-packages (from datasets->easy-tpp) (0.70.16)\n",
      "Requirement already satisfied: fsspec<=2024.12.0,>=2023.1.0 in /usr/local/lib/python3.11/dist-packages (from fsspec[http]<=2024.12.0,>=2023.1.0->datasets->easy-tpp) (2024.12.0)\n",
      "Requirement already satisfied: aiohttp in /usr/local/lib/python3.11/dist-packages (from datasets->easy-tpp) (3.11.15)\n",
      "Requirement already satisfied: huggingface-hub>=0.24.0 in /usr/local/lib/python3.11/dist-packages (from datasets->easy-tpp) (0.30.1)\n",
      "Requirement already satisfied: antlr4-python3-runtime==4.9.* in /usr/local/lib/python3.11/dist-packages (from omegaconf->easy-tpp) (4.9.3)\n",
      "Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.11/dist-packages (from pandas->easy-tpp) (2.8.2)\n",
      "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.11/dist-packages (from pandas->easy-tpp) (2025.2)\n",
      "Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.11/dist-packages (from pandas->easy-tpp) (2025.2)\n",
      "Requirement already satisfied: absl-py>=0.4 in /usr/local/lib/python3.11/dist-packages (from tensorboard->easy-tpp) (1.4.0)\n",
      "Requirement already satisfied: grpcio>=1.48.2 in /usr/local/lib/python3.11/dist-packages (from tensorboard->easy-tpp) (1.71.0)\n",
      "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.11/dist-packages (from tensorboard->easy-tpp) (3.7)\n",
      "Requirement already satisfied: protobuf!=4.24.0,>=3.19.6 in /usr/local/lib/python3.11/dist-packages (from tensorboard->easy-tpp) (5.29.4)\n",
      "Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.11/dist-packages (from tensorboard->easy-tpp) (75.2.0)\n",
      "Requirement already satisfied: six>1.9 in /usr/local/lib/python3.11/dist-packages (from tensorboard->easy-tpp) (1.17.0)\n",
      "Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /usr/local/lib/python3.11/dist-packages (from tensorboard->easy-tpp) (0.7.2)\n",
      "Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.11/dist-packages (from tensorboard->easy-tpp) (3.1.3)\n",
      "Requirement already satisfied: typing-extensions>=4.10.0 in /usr/local/lib/python3.11/dist-packages (from torch->easy-tpp) (4.13.0)\n",
      "Requirement already satisfied: networkx in /usr/local/lib/python3.11/dist-packages (from torch->easy-tpp) (3.4.2)\n",
      "Requirement already satisfied: jinja2 in /usr/local/lib/python3.11/dist-packages (from torch->easy-tpp) (3.1.6)\n",
      "Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.4.127 in /usr/local/lib/python3.11/dist-packages (from torch->easy-tpp) (12.4.127)\n",
      "Requirement already satisfied: nvidia-cuda-runtime-cu12==12.4.127 in /usr/local/lib/python3.11/dist-packages (from torch->easy-tpp) (12.4.127)\n",
      "Requirement already satisfied: nvidia-cuda-cupti-cu12==12.4.127 in /usr/local/lib/python3.11/dist-packages (from torch->easy-tpp) (12.4.127)\n",
      "Requirement already satisfied: nvidia-cudnn-cu12==9.1.0.70 in /usr/local/lib/python3.11/dist-packages (from torch->easy-tpp) (9.1.0.70)\n",
      "Requirement already satisfied: nvidia-cublas-cu12==12.4.5.8 in /usr/local/lib/python3.11/dist-packages (from torch->easy-tpp) (12.4.5.8)\n",
      "Requirement already satisfied: nvidia-cufft-cu12==11.2.1.3 in /usr/local/lib/python3.11/dist-packages (from torch->easy-tpp) (11.2.1.3)\n",
      "Requirement already satisfied: nvidia-curand-cu12==10.3.5.147 in /usr/local/lib/python3.11/dist-packages (from torch->easy-tpp) (10.3.5.147)\n",
      "Requirement already satisfied: nvidia-cusolver-cu12==11.6.1.9 in /usr/local/lib/python3.11/dist-packages (from torch->easy-tpp) (11.6.1.9)\n",
      "Requirement already satisfied: nvidia-cusparse-cu12==12.3.1.170 in /usr/local/lib/python3.11/dist-packages (from torch->easy-tpp) (12.3.1.170)\n",
      "Requirement already satisfied: nvidia-cusparselt-cu12==0.6.2 in /usr/local/lib/python3.11/dist-packages (from torch->easy-tpp) (0.6.2)\n",
      "Requirement already satisfied: nvidia-nccl-cu12==2.21.5 in /usr/local/lib/python3.11/dist-packages (from torch->easy-tpp) (2.21.5)\n",
      "Requirement already satisfied: nvidia-nvtx-cu12==12.4.127 in /usr/local/lib/python3.11/dist-packages (from torch->easy-tpp) (12.4.127)\n",
      "Requirement already satisfied: nvidia-nvjitlink-cu12==12.4.127 in /usr/local/lib/python3.11/dist-packages (from torch->easy-tpp) (12.4.127)\n",
      "Requirement already satisfied: triton==3.2.0 in /usr/local/lib/python3.11/dist-packages (from torch->easy-tpp) (3.2.0)\n",
      "Requirement already satisfied: sympy==1.13.1 in /usr/local/lib/python3.11/dist-packages (from torch->easy-tpp) (1.13.1)\n",
      "Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.11/dist-packages (from sympy==1.13.1->torch->easy-tpp) (1.3.0)\n",
      "Requirement already satisfied: aiohappyeyeballs>=2.3.0 in /usr/local/lib/python3.11/dist-packages (from aiohttp->datasets->easy-tpp) (2.6.1)\n",
      "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.11/dist-packages (from aiohttp->datasets->easy-tpp) (1.3.2)\n",
      "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.11/dist-packages (from aiohttp->datasets->easy-tpp) (25.3.0)\n",
      "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.11/dist-packages (from aiohttp->datasets->easy-tpp) (1.5.0)\n",
      "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.11/dist-packages (from aiohttp->datasets->easy-tpp) (6.3.1)\n",
      "Requirement already satisfied: propcache>=0.2.0 in /usr/local/lib/python3.11/dist-packages (from aiohttp->datasets->easy-tpp) (0.3.1)\n",
      "Requirement already satisfied: yarl<2.0,>=1.17.0 in /usr/local/lib/python3.11/dist-packages (from aiohttp->datasets->easy-tpp) (1.18.3)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.11/dist-packages (from requests>=2.32.2->datasets->easy-tpp) (3.4.1)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.11/dist-packages (from requests>=2.32.2->datasets->easy-tpp) (3.10)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.11/dist-packages (from requests>=2.32.2->datasets->easy-tpp) (2.3.0)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.11/dist-packages (from requests>=2.32.2->datasets->easy-tpp) (2025.1.31)\n",
      "Requirement already satisfied: MarkupSafe>=2.1.1 in /usr/local/lib/python3.11/dist-packages (from werkzeug>=1.0.1->tensorboard->easy-tpp) (3.0.2)\n",
      "Downloading easy_tpp-0.1.2-py3-none-any.whl (126 kB)\n",
      "\u001B[2K   \u001B[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001B[0m \u001B[32m126.5/126.5 kB\u001B[0m \u001B[31m2.9 MB/s\u001B[0m eta \u001B[36m0:00:00\u001B[0m\n",
      "\u001B[?25hInstalling collected packages: easy-tpp\n",
      "  Attempting uninstall: easy-tpp\n",
      "    Found existing installation: easy-tpp 0.1.0\n",
      "    Uninstalling easy-tpp-0.1.0:\n",
      "      Successfully uninstalled easy-tpp-0.1.0\n",
      "Successfully installed easy-tpp-0.1.2\n"
     ]
    },
    {
     "output_type": "display_data",
     "data": {
      "application/vnd.colab-display-data+json": {
       "pip_warning": {
        "packages": [
         "easy_tpp"
        ]
       },
       "id": "a33b6aa166cb43cfa25793118ff1aafc"
      }
     },
     "metadata": {}
    }
   ],
   "source": [
    "! pip install --upgrade easy-tpp"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Step 1: setup the config file\n",
    "\n",
    "The EasyTPP framework uses a YAML-based configuration system that consists of two main components:\n",
    "- **data configuration**, which specifies the data sources and their formats. It defines where to find the training, validation, and test datasets (see below as an example). The full explanation of the dataset can be found in the previous tutorial [EasyTPP-Dataset](https://github.com/ant-research/EasyTemporalPointProcess/blob/main/notebooks/easytpp_1_dataset.ipynb).\n",
    "- **model configuration**, which defines the model architecture, hyperparameters, and training settings.\n",
    "\n",
    "Let's first look at the data configuration:"
   ],
   "metadata": {
    "id": "iz34tMDNg80K"
   }
  },
  {
   "cell_type": "code",
   "source": [
    "data_config = \"\"\"\n",
    "data:\n",
    "  taxi:\n",
    "    data_format: json\n",
    "    train_dir:  ./data/taxi/train.json\n",
    "    valid_dir:  ./data/taxi/dev.json\n",
    "    test_dir:  ./data/taxi/test.json\n",
    "\n",
    "\"\"\""
   ],
   "metadata": {
    "id": "3Sap8WlqgNQz"
   },
   "execution_count": 1,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "The model configuration specifies the neural network architecture and training hyperparameters. It is structured into two main sections: base configuration for general settings and trainer-specific configuration for detailed training parameters:\n"
   ],
   "metadata": {
    "id": "ZJTYO2u02q6h"
   }
  },
  {
   "cell_type": "code",
   "source": [
    "model_config = \"\"\"\n",
    "NHP_train:\n",
    "  base_config:\n",
    "    stage: train\n",
    "    backend: torch\n",
    "    dataset_id: taxi\n",
    "    runner_id: std_tpp\n",
    "    model_id: NHP # model name\n",
    "    base_dir: './checkpoints/'\n",
    "  trainer_config:\n",
    "    batch_size: 256\n",
    "    max_epoch: 2\n",
    "    shuffle: True\n",
    "    optimizer: adam\n",
    "    learning_rate: 1.e-3\n",
    "    valid_freq: 1\n",
    "    use_tfb: False\n",
    "    metrics: [ 'acc', 'rmse' ]\n",
    "    seed: 2019\n",
    "    gpu: -1\n",
    "\"\"\""
   ],
   "metadata": {
    "id": "Yq9hnfrAzXxI"
   },
   "execution_count": 2,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "As an illustrative example, we write the YAML content to a file yaml_content as below. Note that `runner_config` is the pipeline configuration ID that tells EasyTPP which configuration to use. We combine the data configuration\n",
    "and model configuration into a single YAML file. This file will be used to initialize the model runner, which will handle the training process based on the specified parameters."
   ],
   "metadata": {
    "id": "ZhppAAZ-2xMN"
   }
  },
  {
   "cell_type": "code",
   "source": [
    "yaml_content = \"\"\"\n",
    "pipeline_config_id: runner_config\n",
    "\n",
    "data:\n",
    "    taxi:\n",
    "        data_format: json\n",
    "        train_dir: easytpp/taxi  # ./data/taxi/train.json\n",
    "        valid_dir: easytpp/taxi  # ./data/taxi/dev.json\n",
    "        test_dir: easytpp/taxi   # ./data/taxi/test.json\n",
    "        data_specs:\n",
    "          num_event_types: 10\n",
    "          pad_token_id: 10\n",
    "          padding_side: right\n",
    "\n",
    "\n",
    "NHP_train:\n",
    "  base_config:\n",
    "    stage: train\n",
    "    backend: torch\n",
    "    dataset_id: taxi\n",
    "    runner_id: std_tpp\n",
    "    model_id: NHP # model name\n",
    "    base_dir: './checkpoints/'\n",
    "  trainer_config:\n",
    "    batch_size: 256\n",
    "    max_epoch: 2\n",
    "    shuffle: False\n",
    "    optimizer: adam\n",
    "    learning_rate: 1.e-3\n",
    "    valid_freq: 1\n",
    "    use_tfb: True\n",
    "    metrics: [ 'acc', 'rmse' ]\n",
    "    seed: 2019\n",
    "    gpu: -1\n",
    "  model_config:\n",
    "    hidden_size: 32\n",
    "    loss_integral_num_sample_per_step: 20\n",
    "    thinning:\n",
    "      num_seq: 10\n",
    "      num_sample: 1\n",
    "      num_exp: 500 # number of i.i.d. Exp(intensity_bound) draws at one time in thinning algorithm\n",
    "      look_ahead_time: 10\n",
    "      patience_counter: 5 # the maximum iteration used in adaptive thinning\n",
    "      over_sample_rate: 5\n",
    "      num_samples_boundary: 5\n",
    "      dtime_max: 5\n",
    "      num_step_gen: 1\n",
    "\"\"\"\n",
    "\n",
    "# Save the content to a file named config.yaml\n",
    "with open(\"config.yaml\", \"w\") as file:\n",
    "    file.write(yaml_content)"
   ],
   "metadata": {
    "id": "BZk6_2092xrD"
   },
   "execution_count": 3,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Step 2: Train the model\n",
    "\n",
    "Then we can initialize the model runner using the configuration file we just created. The model runner will handle the training process based on the specified parameters in our configuration.\n",
    "\n",
    "We'll use the `Config` class to build the configuration from the YAML file and specify an experiment ID (key of the `model_config`). Then, we'll create a `Runner` instance from this configuration and run the training process."
   ],
   "metadata": {
    "id": "tWzAAQrk25zZ"
   }
  },
  {
   "cell_type": "code",
   "source": [
    "from easy_tpp.config_factory import Config\n",
    "from easy_tpp.runner import Runner\n",
    "\n",
    "config = Config.build_from_yaml_file('./config.yaml', experiment_id='NHP_train')\n",
    "\n",
    "model_runner = Runner.build_from_config(config)\n",
    "\n",
    "model_runner.run()"
   ],
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "mkwjV7hL21Q6",
    "outputId": "b5002b2f-ccf3-4267-c009-2848d679d311"
   },
   "execution_count": 4,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "\u001B[31;1m2025-04-06 05:42:18,953 - config.py[pid:60211;line:34:build_from_yaml_file] - CRITICAL: Load pipeline config class RunnerConfig\u001B[0m\n",
      "\u001B[31;1m2025-04-06 05:42:18,957 - runner_config.py[pid:60211;line:151:update_config] - CRITICAL: train model NHP using CPU with torch backend\u001B[0m\n",
      "\u001B[38;20m2025-04-06 05:42:18,970 - runner_config.py[pid:60211;line:35:__init__] - INFO: Save the config to ./checkpoints/60211_133800770625536_250406-054218/NHP_train_output.yaml\u001B[0m\n",
      "\u001B[38;20m2025-04-06 05:42:18,972 - base_runner.py[pid:60211;line:176:save_log] - INFO: Save the log to ./checkpoints/60211_133800770625536_250406-054218/log\u001B[0m\n"
     ]
    },
    {
     "output_type": "stream",
     "name": "stderr",
     "text": [
      "/usr/local/lib/python3.11/dist-packages/huggingface_hub/utils/_auth.py:94: UserWarning: \n",
      "The secret `HF_TOKEN` does not exist in your Colab secrets.\n",
      "To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.\n",
      "You will be able to reuse this secret in all of your notebooks.\n",
      "Please note that authentication is recommended but still optional to access public models or datasets.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "0.2244252199397379 0.29228809611195583\n",
      "min_dt: 0.000277777777777\n",
      "max_dt: 5.721388888888889\n"
     ]
    },
    {
     "output_type": "stream",
     "name": "stderr",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.11/dist-packages/tensorflow/python/compat/v2_compat.py:98: disable_resource_variables (from tensorflow.python.ops.resource_variables_toggle) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "non-resource variables are not supported in the long term\n"
     ]
    },
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "\u001B[38;20m2025-04-06 05:42:34,617 - tpp_runner.py[pid:60211;line:60:_init_model] - INFO: Num of model parameters 15252\u001B[0m\n",
      "\u001B[38;20m2025-04-06 05:42:36,901 - base_runner.py[pid:60211;line:98:train] - INFO: Data 'taxi' loaded...\u001B[0m\n",
      "\u001B[38;20m2025-04-06 05:42:36,902 - base_runner.py[pid:60211;line:103:train] - INFO: Start NHP training...\u001B[0m\n",
      "\u001B[38;20m2025-04-06 05:42:39,574 - tpp_runner.py[pid:60211;line:96:_train_model] - INFO: [ Epoch 0 (train) ]: train loglike is -1.755373397054743, num_events is 50454\u001B[0m\n",
      "\u001B[38;20m2025-04-06 05:42:45,689 - tpp_runner.py[pid:60211;line:107:_train_model] - INFO: [ Epoch 0 (valid) ]:  valid loglike is -1.6691416010202664, num_events is 7204, acc is 0.4415602443087174, rmse is 0.33315836060539783\u001B[0m\n",
      "\u001B[38;20m2025-04-06 05:42:54,655 - tpp_runner.py[pid:60211;line:122:_train_model] - INFO: [ Epoch 0 (test) ]: test loglike is -1.6577474861303745, num_events is 14420, acc is 0.4467406380027739, rmse is 0.34015134195006963\u001B[0m\n",
      "\u001B[31;1m2025-04-06 05:42:54,657 - tpp_runner.py[pid:60211;line:124:_train_model] - CRITICAL: current best loglike on valid set is -1.6691 (updated at epoch-0), best updated at this epoch\u001B[0m\n",
      "\u001B[38;20m2025-04-06 05:42:57,202 - tpp_runner.py[pid:60211;line:96:_train_model] - INFO: [ Epoch 1 (train) ]: train loglike is -1.6284447567648255, num_events is 50454\u001B[0m\n",
      "\u001B[38;20m2025-04-06 05:43:01,056 - tpp_runner.py[pid:60211;line:107:_train_model] - INFO: [ Epoch 1 (valid) ]:  valid loglike is -1.5259201159945863, num_events is 7204, acc is 0.4582176568573015, rmse is 0.3376860494138715\u001B[0m\n",
      "\u001B[38;20m2025-04-06 05:43:10,024 - tpp_runner.py[pid:60211;line:122:_train_model] - INFO: [ Epoch 1 (test) ]: test loglike is -1.5121817029299585, num_events is 14420, acc is 0.4597087378640777, rmse is 0.34172900829909414\u001B[0m\n",
      "\u001B[31;1m2025-04-06 05:43:10,025 - tpp_runner.py[pid:60211;line:124:_train_model] - CRITICAL: current best loglike on valid set is -1.5259 (updated at epoch-1), best updated at this epoch\u001B[0m\n",
      "\u001B[38;20m2025-04-06 05:43:10,028 - base_runner.py[pid:60211;line:110:train] - INFO: End NHP train! Cost time: 0.552m\u001B[0m\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Step 2: Evaluate a model\n",
    "\n",
    "After training completes, we can evaluate the model's performance on the test dataset. EasyTPP offers streamlined methods for loading trained models and conducting evaluations.\n",
    "\n",
    "First, we need to locate the saved model directory within the `checkpoints` folder specified in our configuration. For example, the model might be stored at `./checkpoints/60211_133800770625536_250406-054218/models/saved_model`.\n",
    "\n",
    "Next, we'll create a new configuration file specifically for evaluation. This evaluation process performs one-step prediction on the validation set defined in the `valid_dir` parameter.\n",
    "\n",
    "For demonstration purposes, we'll use the same dataset for our evaluation as we used during training.\n",
    "\n",
    "We write the YAML config for evaluation as below. This configuration specifies:\n",
    "- The data sources for training, validation, and testing (only validation will be used, the other two are optional).\n",
    "- The evaluation stage and backend framework\n",
    "- Model parameters and batch size\n",
    "- The path to our pretrained model from the previous training step\n",
    "\n",
    "Note: Make sure the `thinning_config` is included in the configuration. Also, ensure that 'rmse' and 'acc' are specified in the `metrics` list under `trainer_config` to properly compute and return these evaluation metrics. The error in the previous run was due to missing 'rmse' in the metrics configuration."
   ],
   "metadata": {
    "id": "66aUobRvJktc"
   }
  },
  {
   "cell_type": "code",
   "source": [
    "eval_yaml =\"\"\"\n",
    "pipeline_config_id: runner_config\n",
    "\n",
    "data:\n",
    "  taxi:\n",
    "    data_format: json\n",
    "    train_dir: easytpp/taxi  # ./data/taxi/train.json\n",
    "    valid_dir: easytpp/taxi  # ./data/taxi/dev.json\n",
    "    test_dir: easytpp/taxi   # ./data/taxi/test.json\n",
    "    data_specs:\n",
    "      num_event_types: 10\n",
    "      pad_token_id: 10\n",
    "      padding_side: right\n",
    "\n",
    "NHP_eval:\n",
    "  base_config:\n",
    "    stage: eval\n",
    "    backend: torch\n",
    "    dataset_id: taxi\n",
    "    runner_id: std_tpp\n",
    "    base_dir: './checkpoints/'\n",
    "    model_id: NHP\n",
    "  trainer_config:\n",
    "    batch_size: 256\n",
    "    max_epoch: 1\n",
    "    metrics: [ 'acc', 'rmse' ]\n",
    "  model_config:\n",
    "    hidden_size: 32\n",
    "    use_ln: False\n",
    "    seed: 2019\n",
    "    gpu: -1\n",
    "    pretrained_model_dir: ./checkpoints/60211_133800770625536_250406-054218/models/saved_model\n",
    "    thinning:\n",
    "      num_seq: 10\n",
    "      num_sample: 1\n",
    "      num_exp: 500 # number of i.i.d. Exp(intensity_bound) draws at one time in thinning algorithm\n",
    "      look_ahead_time: 10\n",
    "      patience_counter: 5 # the maximum iteration used in adaptive thinning\n",
    "      over_sample_rate: 5\n",
    "      num_samples_boundary: 5\n",
    "      dtime_max: 5\n",
    "\"\"\"\n",
    "\n",
    "# Save the content to a file named config.yaml\n",
    "with open(\"eval_config.yaml\", \"w\") as file:\n",
    "    file.write(eval_yaml)"
   ],
   "metadata": {
    "id": "8ADVVKg1JlG1"
   },
   "execution_count": 6,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "We use the following script to evaluate the trained model. This will load the pretrained model from the specified directory and evaluate it on the test dataset\n",
    ""
   ],
   "metadata": {
    "id": "32WAmhBrClUD"
   }
  },
  {
   "cell_type": "code",
   "source": [
    "from easy_tpp.config_factory import Config\n",
    "from easy_tpp.runner import Runner\n",
    "\n",
    "config = Config.build_from_yaml_file('./eval_config.yaml', experiment_id='NHP_eval')\n",
    "\n",
    "model_runner = Runner.build_from_config(config)\n",
    "\n",
    "model_runner.run()"
   ],
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "bwPC6DI9J2LT",
    "outputId": "08030865-15eb-49f8-c2a7-f53363cac43d"
   },
   "execution_count": 7,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "\u001B[31;1m2025-04-06 05:43:28,529 - config.py[pid:60211;line:34:build_from_yaml_file] - CRITICAL: Load pipeline config class RunnerConfig\u001B[0m\n",
      "\u001B[31;1m2025-04-06 05:43:28,532 - runner_config.py[pid:60211;line:151:update_config] - CRITICAL: validate model NHP using CPU with torch backend\u001B[0m\n",
      "\u001B[38;20m2025-04-06 05:43:28,543 - runner_config.py[pid:60211;line:35:__init__] - INFO: Save the config to ./checkpoints/60211_133800770625536_250406-054328/NHP_eval_output.yaml\u001B[0m\n",
      "\u001B[38;20m2025-04-06 05:43:28,545 - base_runner.py[pid:60211;line:176:save_log] - INFO: Save the log to ./checkpoints/60211_133800770625536_250406-054328/log\u001B[0m\n",
      "0.2244252199397379 0.29228809611195583\n",
      "min_dt: 0.000277777777777\n",
      "max_dt: 5.721388888888889\n",
      "\u001B[38;20m2025-04-06 05:43:29,769 - tpp_runner.py[pid:60211;line:60:_init_model] - INFO: Num of model parameters 15252\u001B[0m\n",
      "\u001B[31;1m2025-04-06 05:43:29,773 - tpp_runner.py[pid:60211;line:81:_load_model] - CRITICAL: Load model from ./checkpoints/60211_133800770625536_250406-054218/models/saved_model\u001B[0m\n",
      "\u001B[38;20m2025-04-06 05:43:30,305 - base_runner.py[pid:60211;line:117:evaluate] - INFO: Data 'taxi' loaded...\u001B[0m\n",
      "\u001B[38;20m2025-04-06 05:43:30,306 - base_runner.py[pid:60211;line:122:evaluate] - INFO: Start NHP evaluation...\u001B[0m\n",
      "\u001B[31;1m2025-04-06 05:43:35,132 - tpp_runner.py[pid:60211;line:148:_evaluate_model] - CRITICAL: Evaluation result: loglike is -1.5259201159945863, num_events is 7204, acc is 0.4583564686285397, rmse is 0.3325426308842327\u001B[0m\n",
      "\u001B[38;20m2025-04-06 05:43:35,133 - base_runner.py[pid:60211;line:128:evaluate] - INFO: End NHP evaluation! Cost time: 0.080m\u001B[0m\n"
     ]
    },
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "np.float64(0.3325426308842327)"
      ]
     },
     "metadata": {},
     "execution_count": 7
    }
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Step 3: Generate predictions\n",
    "\n",
    "After training and evaluation, we can generate predictions for new events given a sequence as `context`.\n",
    "\n",
    "The generation process uses the trained model to perform the one-step (by default) or multi-step prediction.\n",
    "\n",
    "In order to evaluate the prediction accuracy, we automatically mask the last-n events (where n depends on the prediction step count) and use them as ground truth (golden events) to compare against our model's predictions.\n",
    "\n",
    "We'll use the same configuration file we used for evaluation, but we'll specify a different experiment ID for the prediction task.\n",
    "\n"
   ],
   "metadata": {
    "id": "ewCrMYQ0HxLU"
   }
  },
  {
   "cell_type": "code",
   "source": [
    "gen_yaml =\"\"\"\n",
    "pipeline_config_id: runner_config\n",
    "\n",
    "data:\n",
    "  taxi:\n",
    "    data_format: json\n",
    "    train_dir: easytpp/taxi  # ./data/taxi/train.json\n",
    "    valid_dir: easytpp/taxi  # ./data/taxi/dev.json\n",
    "    test_dir: easytpp/taxi   # ./data/taxi/test.json\n",
    "    data_specs:\n",
    "      num_event_types: 10\n",
    "      pad_token_id: 10\n",
    "      padding_side: right\n",
    "\n",
    "NHP_gen:\n",
    "  base_config:\n",
    "    stage: gen\n",
    "    backend: torch\n",
    "    dataset_id: taxi\n",
    "    runner_id: std_tpp\n",
    "    base_dir: './checkpoints/'\n",
    "    model_id: NHP\n",
    "  trainer_config:\n",
    "    batch_size: 256\n",
    "    max_epoch: 1\n",
    "    metrics: [ 'acc', 'rmse' ]\n",
    "  model_config:\n",
    "    hidden_size: 32\n",
    "    use_ln: False\n",
    "    seed: 2019\n",
    "    gpu: -1\n",
    "    pretrained_model_dir: ./checkpoints/60211_133800770625536_250406-054218/models/saved_model\n",
    "    thinning:\n",
    "      num_seq: 10\n",
    "      num_sample: 1\n",
    "      num_exp: 500 # number of i.i.d. Exp(intensity_bound) draws at one time in thinning algorithm\n",
    "      look_ahead_time: 10\n",
    "      patience_counter: 5 # the maximum iteration used in adaptive thinning\n",
    "      over_sample_rate: 5\n",
    "      num_samples_boundary: 5\n",
    "      dtime_max: 5\n",
    "      num_step_gen: 1\n",
    "\"\"\"\n",
    "\n",
    "# Save the content to a file named config.yaml\n",
    "with open(\"gen_config.yaml\", \"w\") as file:\n",
    "    file.write(gen_yaml)"
   ],
   "metadata": {
    "id": "FIGDkivPLO3N"
   },
   "execution_count": 8,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "We continue to use a similar configuration file to launch the generation process. Note that it is necessary to explicitly specify `num_gen_step` in the thinning config. The results will be saved in a pickle file.\n",
    "\n",
    "We acknowledge that the generation pipeline is still a work in progress. For instance, the multi-step sampling process can be further improved (ongoing work), and the output format could be enhanced to facilitate easier evaluation."
   ],
   "metadata": {
    "id": "ymXwak6WoOGB"
   }
  },
  {
   "cell_type": "code",
   "source": [
    "from easy_tpp.config_factory import Config\n",
    "from easy_tpp.runner import Runner\n",
    "\n",
    "config = Config.build_from_yaml_file('./gen_config.yaml', experiment_id='NHP_gen')\n",
    "\n",
    "model_runner = Runner.build_from_config(config)\n",
    "\n",
    "model_runner.run()"
   ],
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "kgJVjrh9HisJ",
    "outputId": "f62fe3bc-8484-49f5-f814-166971d2a27a"
   },
   "execution_count": 9,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "\u001B[31;1m2025-04-06 05:43:48,495 - config.py[pid:60211;line:34:build_from_yaml_file] - CRITICAL: Load pipeline config class RunnerConfig\u001B[0m\n",
      "\u001B[31;1m2025-04-06 05:43:48,498 - runner_config.py[pid:60211;line:151:update_config] - CRITICAL: predict model NHP using CPU with torch backend\u001B[0m\n",
      "\u001B[38;20m2025-04-06 05:43:48,508 - runner_config.py[pid:60211;line:35:__init__] - INFO: Save the config to ./checkpoints/60211_133800770625536_250406-054348/NHP_gen_output.yaml\u001B[0m\n",
      "\u001B[38;20m2025-04-06 05:43:48,510 - base_runner.py[pid:60211;line:176:save_log] - INFO: Save the log to ./checkpoints/60211_133800770625536_250406-054348/log\u001B[0m\n",
      "0.2244252199397379 0.29228809611195583\n",
      "min_dt: 0.000277777777777\n",
      "max_dt: 5.721388888888889\n",
      "\u001B[38;20m2025-04-06 05:43:49,245 - tpp_runner.py[pid:60211;line:60:_init_model] - INFO: Num of model parameters 15252\u001B[0m\n",
      "\u001B[31;1m2025-04-06 05:43:49,249 - tpp_runner.py[pid:60211;line:81:_load_model] - CRITICAL: Load model from ./checkpoints/60211_133800770625536_250406-054218/models/saved_model\u001B[0m\n",
      "\u001B[38;20m2025-04-06 05:43:49,777 - base_runner.py[pid:60211;line:135:gen] - INFO: Data 'taxi' loaded...\u001B[0m\n",
      "\u001B[38;20m2025-04-06 05:43:49,779 - base_runner.py[pid:60211;line:140:gen] - INFO: Start NHP evaluation...\u001B[0m\n",
      "\u001B[31;1m2025-04-06 05:43:50,159 - tpp_runner.py[pid:60211;line:162:_gen_model] - CRITICAL: Save the prediction to pickle file pred.pkl\u001B[0m\n",
      "\u001B[38;20m2025-04-06 05:43:50,161 - base_runner.py[pid:60211;line:146:gen] - INFO: End NHP generation! Cost time: 0.006m\u001B[0m\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "source": [],
   "metadata": {
    "id": "pTreVIxWof_R"
   }
  }
 ]
}