{
  "created_at": "2026-05-30T14:42:46.285459",
  "source_checkpoint": "/mnt/d/Development/Libraries/Mistral_Nemo_Instruct_2407",
  "output_path": "/mnt/d/Development/Libraries/Mistral-Nemo-Instruct-2407-AWQ_20260529_152601",
  "awq": {
    "scheme": "W4A16_ASYM",
    "targets": [
      "Linear"
    ],
    "ignore": [
      "lm_head"
    ],
    "duo_scaling": "both",
    "save_compressed": true,
    "safe_serialization": true
  },
  "calibration": {
    "source": "hf_dataset",
    "num_samples_requested": 256,
    "num_samples_loaded": 256,
    "max_seq_length": 512
  },
  "timing": {
    "compression_seconds": 81080.89208602905,
    "compression_minutes": 1351.3482014338176
  },
  "memory": {
    "peak_gpu_allocated_gib": 29.476911544799805
  },
  "validation": [
    {
      "prompt": "Explain AWQ quantization in one paragraph.",
      "output": "Explain AWQ quantization in one paragraph. AWQ (Adaptive Weight Quantization) is a technique used in machine learning model compression to reduce the model size and improve inference speed. It adaptively quantizes the weights of the model based on their sensitivity, preserving important weights and coarsely quantizing less important ones. This results in a significant reduction in model size and computational cost, while maintaining a high level of model accuracy."
    },
    {
      "prompt": "Return a compact JSON object with keys model, format, and runtime.",
      "output": "Return a compact JSON object with keys model, format, and runtime. The model key should contain the name of the model, the format key should contain the format of the model, and the runtime key should contain the runtime of the model. The model name should be \"my_model\", the format should be \"json\", and the runtime should be \"1.23456789\".\n\nHere is a Python solution:\n\n```python\nimport json\n\nmodel = \"my_model\"\nformat = \"json\"\nruntime = 1.23456789\n\ndata = {\n    \"model\": model,\n    \"format\": format,\n    \"runtime\": runtime\n}\n\nprint(json"
    },
    {
      "prompt": "Why does INT4 weight-only quantization help batched inference?",
      "output": "Why does INT4 weight-only quantization help batched inference? I am trying to understand the motivation behind the paper \"Quantizing Neural Networks for On-Device Vision Applications\" by Google. Here is the link to the paper: https://arxiv.org/abs/1712.05877\n\nIn the paper, they mention that INT4 weight-only quantization helps batched inference. I am trying to understand why this is the case. I understand that INT4 quantization reduces the memory footprint and computational cost, but I am not sure why it helps batched inference specifically.\n\nCan someone please explain this to me? Thank you!\n\n## Answer (1)\n\nThe reason is that"
    }
  ],
  "runtime_target": "vLLM batched inference"
}