{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LITA Checkpoint Conversion, Finetuning and Inference Tutorial"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Note:\n",
    "Currently, this notebook can be run in a NeMo container (>= 24.07). An example command to launch the container:\n",
    "\n",
    "```\n",
    "docker run --gpus all -it --rm  -v $PWD:/ws --shm-size=8g -p 8888:8888 --ulimit memlock=-1 --ulimit stack=67108864 <your_nemo_container>\n",
    "```\n",
    "For inference and finetuning, you need to increase the share memory size to avoid some OOM issue. For example,\n",
    "```\n",
    "docker run --gpus all -it --rm  -v $PWD:/ws --shm-size=128g -p 8888:8888 --ulimit memlock=-1 --ulimit stack=67108864 nvcr.io/nvidia/nemo:dev\n",
    "```\n",
    "\n",
    "By `-v $PWD:/ws`, we can mount the current local directory to `/ws/` in docker container. We may use this local directory to put the `NeMo` source code, checkpoints and dataset we will generate.\n",
    "\n",
    "If you wanna use NeMo container (>24.04 and < 24.07) (not recommended), you need to manually mount the latest nemo:\n",
    "```\n",
    "docker run --gpus all -it --rm -v <your_nemo_dir>:/opt/NeMo -v $PWD:/ws --shm-size=128g -p 8888:8888 --ulimit memlock=-1 --ulimit stack=67108864 <your_nemo_container>\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LITA Introduction\n",
    "\n",
    "[LITA](https://arxiv.org/pdf/2403.19046) stands for Language Instructed Temporal-Localization Assistant, which demonstrates strong performance on Reasoning Temporal Localization (RTL) task. It introduces time tokens to better help LLM understand 'When?' question in video. The below figure from [LITA paper](https://arxiv.org/pdf/2403.19046) shows a clear idea of how LITA works.\n",
    "\n",
    "<img src=\"images/LITA_arch.png\" alt=\"drawing\" style=\"width:800px;\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenizer and Checkpoint Conversion\n",
    "As we learned that LITA introduces `time tokens` so that timestamps of events in a video would be represented as time tokens instead of the original float point timestamps. Therefore we need to add these time tokens to the tokenizer of the backbone/LLM model. In this example, we take `Llama-3-VILA1.5-8B` as an example to show how to integrate LITA to a LLaVA like model. You may also use similar steps to convert other llama or LLaVA like models that have backbone LLM as llama such as [vicuna](https://huggingface.co/lmsys/vicuna-13b-v1.5) and [llava-v1.6-vicuna-13b](https://huggingface.co/liuhaotian/llava-v1.6-vicuna-13b).\n",
    "\n",
    "Please download the huggingface `Llama-3-VILA1.5-8B` model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "mkdir /ws/pretrained_models && cd /ws/pretrained_models\n",
    "git clone https://huggingface.co/Efficient-Large-Model/Llama-3-VILA1.5-8B"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tokenizer conversion\n",
    "Here we show how to add 100 time tokens and some nemo extra tokens to a huggingface tokenizer.\n",
    "For the definition of nemo extra tokens, please refer to `/opt/NeMo/nemo/collections/multimodal/data/neva/conversation.py`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define the TIME_TOKEN_TEMPLATE\n",
    "TIME_TOKEN_TEMPLATE = \"<t{t}>\"\n",
    "hf_llm_model_path='/ws/pretrained_models/Llama-3-VILA1.5-8B/llm'\n",
    "tokenizer_path = '/ws/converted_models/tokenizer/'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "import transformers\n",
    "tokenizer = transformers.AutoTokenizer.from_pretrained(hf_llm_model_path)\n",
    "DEFAULT_IM_START_TOKEN = \"<extra_id_4>\" # mark the start of the slow token\n",
    "DEFAULT_IM_END_TOKEN = \"<extra_id_5>\" # the end of the slow token\n",
    "VID_START_TOKEN = \"<extra_id_8>\" # the start of the fast token\n",
    "VID_END_TOKEN = \"<extra_id_9>\" # the end of the fast token\n",
    "num_new_tokens = tokenizer.add_tokens([DEFAULT_IM_START_TOKEN, DEFAULT_IM_END_TOKEN, VID_START_TOKEN, VID_END_TOKEN], special_tokens=True)\n",
    "tokenizer.pad_token = tokenizer.eos_token  # use eos token as pad token\n",
    "num_time_tokens = 100\n",
    "time_tokens = [TIME_TOKEN_TEMPLATE.format(t=x) for x in range(num_time_tokens)]\n",
    "num_new_tokens = tokenizer.add_tokens(time_tokens)\n",
    "# add the other nemo extra tokens\n",
    "extra_tokens = [\"<extra_id_0>\",\"<extra_id_1>\",\"<extra_id_2>\",\"<extra_id_3>\",\"<extra_id_6>\",\"<extra_id_7>\"]\n",
    "tokenizer.add_tokens(extra_tokens)\n",
    "tokenizer.save_pretrained(tokenizer_path)\n",
    "print(len(tokenizer.vocab))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can check the tokenizer by:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nemo.collections.nlp.modules.common.tokenizer_utils import get_nmt_tokenizer\n",
    "tokenizer = get_nmt_tokenizer(library=\"huggingface\", model_name=tokenizer_path)\n",
    "print(len(tokenizer.vocab))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice if you wanna convert checkpoints trained from [LITA1.0](https://github.com/NVlabs/LITA), you should put all the extra tokens including `DEFAULT_IM_START_TOKEN` and `DEFAULT_IM_END_TOKEN` at the end of the time tokens."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Checkpoint Conversion\n",
    "Since VILA and LITA shared a similar model structure as LLaVA, we'll leverage `/opt/NeMo/examples/multimodal/multimodal_llm/neva/convert_llava_to_neva.py` for converting the checkpoint. Since VILA and LITA depends on LLaVA, we need to clone LLaVA first.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "git clone --depth 1 --branch v1.2.2 https://github.com/haotian-liu/LLaVA/ /ws/LLaVA\n",
    "cd /ws"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "export PYTHONPATH=/ws/LLaVA:$PYTHONPATH\n",
    "# check the config file in /opt/NeMo/examples/multimodal/multimodal_llm/neva/conf/vita_config.yaml\n",
    "python /opt/NeMo/examples/multimodal/multimodal_llm/neva/convert_llava_to_neva.py \\\n",
    "    --in-file /ws/pretrained_models/Llama-3-VILA1.5-8B/llm \\\n",
    "    --mm-vision-tower /ws/pretrained_models/Llama-3-VILA1.5-8B/vision_tower \\\n",
    "    --mm-projector-ckpt-dir /ws/pretrained_models/Llama-3-VILA1.5-8B/mm_projector \\\n",
    "    --out-file /ws/converted_models/Llama-3-VILA1.5-8B.nemo \\\n",
    "    --tokenizer-model /ws/converted_models/tokenizer/ \\\n",
    "    --config-file vita_config.yaml \\\n",
    "    --model-type VITA \\\n",
    "    --conv-template llama_3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice if `mm_vision_tower` can be downloaded from huggingface and you don't want to change it, then you don't need to explicitly add this option.  And similarly, only when you want to change the `mm_projector`, you will need to add the `mm_projector_ckpt_dir`.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Finetuning\n",
    "\n",
    "In this section, we'll preprocess the Dense Video Captioning dataset and then do finetuning with the nemo ckpt we just converted."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convert Dataset\n",
    "The targeted dataset file format for finetuning should be like:\n",
    "```bash\n",
    "[\n",
    "    # 1st example: video question answer\n",
    "    {\n",
    "        \"id\": \"1043215450\",\n",
    "        \"video\": \"076101_076150/1043215450.mp4\",   # video_path will be prepended\n",
    "        \"conversations\": \n",
    "        [\n",
    "            {\"from\": \"human\", \"value\": \"<video>\\n is the athlete wearing trousers\"}, \n",
    "            {\"from\": \"gpt\", \"value\": \"Yes\"}\n",
    "        ]       \n",
    "    },\n",
    "    # 2nd example: dense video captioning\n",
    "    {\n",
    "        \"id\": \"xxxx\",\n",
    "        \"video: \"xxxx.mp4\",\n",
    "        \"conversations\":\n",
    "        [\n",
    "            {\"from\": \"human\", \"value\": \"<video>\\n \"Provide a detailed description of the given video.Prepend each sentence with its start and end timestamps.\"}, \n",
    "            {\"from\": \"gpt\", \"value\": \"<t1> <t2> Apply eyeshadow on the crease with brush <t3> <t4> Apply eyeshadow on the outer corner of eyes with brush\"}\n",
    "        ]\n",
    "    },\n",
    "    # 3rd example: event classification\n",
    "    {\n",
    "        \"id\": \"xxxx\",\n",
    "        \"video: \"xxxx.mp4\",\n",
    "        \"conversations\":\n",
    "        [\n",
    "            {\"from\": \"human\", \"value\": \"<video>\\n \"What is the action performed in this video?\"}, \n",
    "            {\"from\": \"gpt\", \"value\": \"brush hair\"}\n",
    "        ]\n",
    "    },\n",
    "    # 4th example: event localization\n",
    "    {\n",
    "        \"id\": \"-4RXOT_UfpM_2\",\n",
    "        \"video\": \"-4RXOT_UfpM_2.mp4\",\n",
    "        \"conversations\": [\n",
    "            {\"from\": \"human\", \"value\": \"<video>\\nWhen is \\\"Apply concealer on the eyelids and blend with sponge\\\" depicted in the video? Provide a response using only start and end timestamps.\"},\n",
    "            {\"from\": \"gpt\", \"value\": \"<t4> <t18>\"}\n",
    "        ],\n",
    "        \"durations\": 119.01901901901903\n",
    "    },\n",
    "    ...\n",
    "]\n",
    "```\n",
    "\n",
    "Here the `<video>` is the placeholder for the video features. In the 2nd example, `<t1>` `<t2>` are the time tokens to indicate in which time interval we've seen this event or description of the time inverval. You can prepare your time tokens like this:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "TIME_TOKEN_TEMPLATE = \"<t{t}>\"\n",
    "def time_to_string(time, num_time_tokens):\n",
    "    max_offset = float(num_time_tokens - 1)\n",
    "    time = int(np.round(max_offset * time))\n",
    "    return TIME_TOKEN_TEMPLATE.format(t=time)\n",
    "\n",
    "# example of converting time tokens\n",
    "# from 10seconds to 15 seconds\n",
    "num_time_tokens = 100\n",
    "start = 10.0   # the 10 seconds\n",
    "end = 15.0     # the 15 seconds\n",
    "duration = 200.0 # total video duration is 200seconds\n",
    "start = start / duration \n",
    "end = end / duration\n",
    "start_time_token_str = time_to_string(start, num_time_tokens)\n",
    "end_time_token_str = time_to_string(end, num_time_tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For Dense Video Captioning (DVC) task or Reasoning Temporal Localization (RTL) task, your dataset probably looks like:\n",
    "```bash\n",
    "{\n",
    "    \"video_name\": {\n",
    "        \"duration\": 125.0,\n",
    "        \"timestamps\": [\n",
    "            [0, 5], \n",
    "            [3, 9]\n",
    "        ],\n",
    "        \"sentences\": [\n",
    "            \"Here is your caption 1\",\n",
    "            \"Here is your caption 2\",\n",
    "        ],\n",
    "        \"events\": [\n",
    "            \"Event 1\",\n",
    "            \"Event 2\",\n",
    "        ]\n",
    "    },\n",
    "    ...\n",
    "}\n",
    "```\n",
    "\n",
    "If you've already prepared this style dataset, you may refer to `convert_dvc_dataset_for_training.py`, `convert_dvc_dataset_for_evaluation.py` and `convert_video_qa_dataset.py` under `/opt/NeMo/scripts/multimodal_dataset_conversion` to convert the datasets so that they could be used in finetuning. If you want to augment your dataset by leveraging the NVIDIA LLM APIs or external LLMs, you may refer to `generate_qa_data.py` under the same directory. For example, suppose your dataset is `train.json`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "# generate custom caption dataset and multiply the dataset by three times\n",
    "python /opt/NeMo/scripts/multimodal_dataset_conversion/convert_dvc_dataset_for_training.py \\\n",
    "    --input_dvc_dataset /ws/dataset/train.json \\\n",
    "    --video_path_prefix /ws/dataset/videos/ \\\n",
    "    --subtask custom_caption --data_multiplier 3 \\\n",
    "    --output_file /ws/dataset/vc_train.json\n",
    "\n",
    "# generate event localization dataset and increase the dataset by three times\n",
    "python /opt/NeMo/scripts/multimodal_dataset_conversion/convert_dvc_dataset_for_training.py \\\n",
    "    --input_dvc_dataset /ws/dataset/train.json \\\n",
    "    --video_path_prefix /ws/dataset/videos/ \\\n",
    "    --subtask event_localization --data_multiplier 3 \\\n",
    "    --output_file /ws/dataset/event_loc_train.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Finetuning\n",
    "We'll use 8xA100(80GB) to do the finetuning. You may set smaller number of `num_frames` or change the `tensor_model_parallel_size` or `pipeline_model_parallel_size` if you encounter OOM issue.\n",
    "\n",
    "Run finetuning by:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "video_folder=/ws/dataset/videos/\n",
    "# You may use vc_train.json or event_loc_train.json in last step or combine them together\n",
    "data_path=/ws/dataset/combined_train.json  # training datasets combining different video tasks;\n",
    "model_path=/ws/converted_models/Llama-3-VILA1.5-8B.nemo\n",
    "EXP_MANAGER_DIR=/ws/train  # check this directory for experiment details\n",
    "num_gpus=8\n",
    "torchrun --nproc_per_node=${num_gpus} /opt/NeMo/examples/multimodal/multimodal_llm/neva/neva_finetune.py \\\n",
    "  --config-path=/opt/NeMo/examples/multimodal/multimodal_llm/neva/conf/ \\\n",
    "  --config-name=vita_config.yaml \\\n",
    "  ++cluster_type=BCP \\\n",
    "  trainer.num_nodes=1 \\\n",
    "  trainer.precision=bf16 \\\n",
    "  trainer.devices=${num_gpus} \\\n",
    "  trainer.max_steps=262 \\\n",
    "  trainer.limit_val_batches=5 \\\n",
    "  model.megatron_amp_O2=false \\\n",
    "  model.mm_cfg.llm.freeze=false \\\n",
    "  model.mm_cfg.vision_encoder.freeze=true \\\n",
    "  model.mm_cfg.vision_encoder.from_pretrained=/ws/pretrained_models/Llama-3-VILA1.5-8B/vision_tower \\\n",
    "  model.mm_cfg.vision_encoder.model_type=siglip_vision_model \\\n",
    "  model.global_batch_size=128 \\\n",
    "  model.micro_batch_size=1 \\\n",
    "  model.tensor_model_parallel_size=4 \\\n",
    "  model.pipeline_model_parallel_size=1 \\\n",
    "  model.restore_from_path=${model_path} \\\n",
    "  model.context_parallel_size=1 \\\n",
    "  model.data.video_folder=${video_folder} \\\n",
    "  model.data.data_path=${data_path} \\\n",
    "  model.data.num_frames=128 \\\n",
    "  model.mm_cfg.use_lita=true \\\n",
    "  model.mm_cfg.lita.lita_video_arch=temporal_all_resolution \\\n",
    "  model.mm_cfg.lita.visual_token_format=im_vid_start_end \\\n",
    "  model.mm_cfg.lita.sample_frames=4 \\\n",
    "  model.mcore_gpt=true \\\n",
    "  model.transformer_engine=true \\\n",
    "  model.optim.sched.warmup_steps=8 \\\n",
    "  exp_manager.create_checkpoint_callback=True \\\n",
    "  exp_manager.create_wandb_logger=False \\\n",
    "  exp_manager.wandb_logger_kwargs.project=neva_lita \\\n",
    "  exp_manager.wandb_logger_kwargs.name=neva_lita_finetuning \\\n",
    "  exp_manager.exp_dir=${EXP_MANAGER_DIR}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation\n",
    "Assume you're trying to do evaluation task on RTL task. Please refer to `/opt/NeMo/scripts/multimodal_dataset_conversion/convert_dvc_dataset_for_evaluation.py` about how to generate RTL task evaluation file from DVC dataset. And assume your evaluation file `rtl_eval.json` is like:\n",
    "\n",
    "```bash\n",
    "[\n",
    "    {\n",
    "        \"video\": \"-4RXOT_UfpM_3.mp4\",\n",
    "        \"question_id\": \"-4RXOT_UfpM_3_0\",\n",
    "        \"question\": \"When does \\\"Apply eyeshadow on the lower area then crease with brush\\\" happen in the video? Provide a response using only start and end timestamps.\",\n",
    "        \"ref_answer\": \"<5> <58> Apply eyeshadow on the lower area then crease with brush\",\n",
    "        \"duration\": 118.01801801801803\n",
    "    },\n",
    "    ...\n",
    "]\n",
    "\n",
    "```\n",
    "\n",
    "Notice the `<5> <58>` are the start and end timestamps (in seconds) of the event.\n",
    "\n",
    "After training, we can split the evaluation file (`rtl_eval.json`) into `$num_gpus` number of input files so that the inference can be accelerated. This is optional.\n",
    "We can do this by:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import json\n",
    "json_file=\"/ws/dataset/valid/rtl_eval.json\"\n",
    "num_splits=8 # suppose you have 8 gpus\n",
    "output_dir=\"/ws/dataset/valid/split/\"\n",
    "os.makedirs(output_dir, exist_ok=True)\n",
    "with open(json_file, 'r') as f:\n",
    "    data = json.load(f)\n",
    "data = sorted(data, key=lambda x: x['video']) # group by video\n",
    "num_samples = len(data)\n",
    "split_size = num_samples // num_splits\n",
    "for i in range(num_splits):\n",
    "    start = i * split_size\n",
    "    end = (i + 1) * split_size if i < num_splits - 1 else num_samples\n",
    "    split_data = data[start:end]\n",
    "    output_file = os.path.join(output_dir, f\"{i}.json\")\n",
    "    with open(output_file, 'w') as f:\n",
    "        json.dump(split_data, f)\n",
    "    print(f\"Saved {end - start} samples to {output_file}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can run the bash script："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "neva_model_file=/ws/train/neva_lita_finetuning.nemo\n",
    "\n",
    "num_gpus=8\n",
    "num_splits=$num_gpus\n",
    "\n",
    "mkdir -p /ws/dataset/valid/split_output\n",
    "\n",
    "for i in $(seq 0 $((num_splits-1)))\n",
    "do\n",
    "    num_gpus=1\n",
    "    # For single input file and single gpu,\n",
    "    # please remove the for loop and set the prompt_file and output_file directly\n",
    "    # prompt_file=/ws/dataset/valid/rtl_eval.json\n",
    "    # output_file=/ws/dataset/valid/nemo_infer_output.json\n",
    "    prompt_file=/ws/dataset/valid/split/$i.json\n",
    "    output_file=/ws/dataset/valid/split_output/nemo_infer_output_$i.json\n",
    "    video_base_path=/ws/dataset/valid/videos/\n",
    "    CUDA_VISIBLE_DEVICES=$i torchrun --nnodes=1 --standalone /opt/NeMo/examples/multimodal/multimodal_llm/neva/neva_evaluation.py \\\n",
    "               --config-path=/opt/NeMo/examples/multimodal/multimodal_llm/neva/conf/ \\\n",
    "               --config-name=neva_inference.yaml \\\n",
    "               tensor_model_parallel_size=1 \\\n",
    "               pipeline_model_parallel_size=1 \\\n",
    "               neva_model_file=$neva_model_file \\\n",
    "               trainer.devices=$num_gpus \\\n",
    "               trainer.precision=bf16 \\\n",
    "               prompt_file=$prompt_file \\\n",
    "               inference.media_base_path=$video_base_path \\\n",
    "               inference.media_type=video \\\n",
    "               output_file=$output_file \\\n",
    "               inference.temperature=0.2 \\\n",
    "               inference.top_k=0 \\\n",
    "               inference.top_p=0.9 \\\n",
    "               inference.greedy=True \\\n",
    "               +add_media_sep=true \\\n",
    "               inference.end_strings='[\"<|eot_id|>\"]' \\\n",
    "               inference.add_BOS=False \\\n",
    "               inference.all_probs=False \\\n",
    "               inference.repetition_penalty=1.2 \\\n",
    "               inference.insert_media_token=left \\\n",
    "               inference.tokens_to_generate=256 \\\n",
    "               +inference.batch_size=8 \\\n",
    "               quantization.algorithm=awq \\\n",
    "               quantization.enable=False \\\n",
    "    &\n",
    "done\n",
    "wait\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# combine the output json files if you split them before\n",
    "input_dir = \"/ws/dataset/valid/split_output/\"\n",
    "output_file = \"/ws/dataset/valid/split_output/nemo_infer_output_total.json\"\n",
    "data = []\n",
    "for file in os.listdir(input_dir):\n",
    "    if file.endswith(\".json\"):\n",
    "        with open(os.path.join(input_dir, file), \"r\") as f:\n",
    "            data.extend(json.load(f))\n",
    "print(f\"Total number of items: {len(data)}\")\n",
    "with open(output_file, \"w\") as f:\n",
    "    json.dump(data, f, indent=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can do the evaluation on RTL task by:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "python3 /opt/NeMo/examples/multimodal/multimodal_llm/neva/eval/eval_video_rtl.py \\\n",
    "    --input_file=/ws/dataset/valid/split_output/nemo_infer_output_total.json \\\n",
    "    --output_dir=/ws/dataset/valid/split_output/ --save_mid_result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `IOU` and `IOU@0.5 precision` metric will be reported. The higher the better.\n",
    "\n",
    "You may also refer to `/opt/NeMo/examples/multimodal/multimodal_llm/neva/eval/eval_vqa.py` to check how to use external LLM API to do the video question answering task evaluation."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}