Spaces:

---

Intel

/

low_bit_open_llm_leaderboard

like 219

Running

Pipeline Failure Report

Model: Quazim0t0/Escarda-86M-Base
Quantization Scheme: MXFP4
Failed Phase: evaluate
Run ID: Escarda-86M-Base-AutoRound-MXFP4-RTN
Error Category: dtype_mismatch

Full Error Log

2026-06-22:14:51:43 INFO     [api.task:312] Building contexts for mmlu_moral_scenarios on rank 0...

  0%|          | 0/895 [00:00<?, ?it/s]
100%|██████████| 895/895 [00:01<00:00, 688.28it/s]
2026-06-22:14:51:44 INFO     [api.task:312] Building contexts for mmlu_philosophy on rank 0...

  0%|          | 0/311 [00:00<?, ?it/s]
100%|██████████| 311/311 [00:00<00:00, 988.25it/s]
2026-06-22:14:51:44 INFO     [api.task:312] Building contexts for mmlu_prehistory on rank 0...

  0%|          | 0/324 [00:00<?, ?it/s]
100%|██████████| 324/324 [00:00<00:00, 973.14it/s]
2026-06-22:14:51:45 INFO     [api.task:312] Building contexts for mmlu_professional_law on rank 0...

  0%|          | 0/1534 [00:00<?, ?it/s]
100%|██████████| 1534/1534 [00:01<00:00, 964.32it/s]
2026-06-22:14:51:46 INFO     [api.task:312] Building contexts for mmlu_world_religions on rank 0...

  0%|          | 0/171 [00:00<?, ?it/s]
100%|██████████| 171/171 [00:00<00:00, 986.62it/s]
2026-06-22:14:51:46 INFO     [api.task:312] Building contexts for hellaswag on rank 0...

  0%|          | 0/10042 [00:00<?, ?it/s]
100%|██████████| 10042/10042 [00:05<00:00, 1675.67it/s]
2026-06-22:14:51:53 INFO     [evaluator:585] Running loglikelihood requests

Tokenizing inputs:   0%|          | 0/100012 [00:00<?, ?it/s]
Tokenizing inputs: 100%|██████████| 100012/100012 [01:56<00:00, 855.09it/s]

Running loglikelihood requests:   0%|          | 0/100012 [00:00<?, ?it/s]Passed argument batch_size = auto:1. Detecting largest batch size
Traceback (most recent call last):
  File "/root/.venv/bin/lm_eval", line 10, in <module>
    sys.exit(cli_evaluate())
             ^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/lm_eval/__main__.py", line 10, in cli_evaluate
    parser.execute(args)
  File "/root/.venv/lib/python3.12/site-packages/lm_eval/_cli/harness.py", line 60, in execute
    args.func(args)
  File "/root/.venv/lib/python3.12/site-packages/lm_eval/_cli/run.py", line 391, in _execute
    results = simple_evaluate(
              ^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/lm_eval/utils.py", line 575, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/lm_eval/evaluator.py", line 358, in simple_evaluate
    results = evaluate(
              ^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/lm_eval/utils.py", line 575, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/lm_eval/evaluator.py", line 596, in evaluate
    resps = getattr(lm, reqtype)(cloned_reqs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/lm_eval/models/huggingface.py", line 1227, in loglikelihood
    return super().loglikelihood(requests, disable_tqdm=disable_tqdm)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/lm_eval/api/model.py", line 446, in loglikelihood
    return self._loglikelihood_tokens(new_reqs, disable_tqdm=disable_tqdm)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/lm_eval/models/huggingface.py", line 1402, in _loglikelihood_tokens
    for chunk in chunks:
                 ^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/lm_eval/models/utils.py", line 315, in get_batched
    yield from batch
  File "/root/.venv/lib/python3.12/site-packages/lm_eval/models/utils.py", line 492, in get_chunks
    if len(arr) == (fn(i, _iter) if fn else n):
                    ^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/lm_eval/models/huggingface.py", line 1328, in _batch_scheduler
    self.batch_sizes[sched] = self._detect_batch_size(n_reordered_requests, pos)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/lm_eval/models/huggingface.py", line 1025, in _detect_batch_size
    batch_size = forward_batch()
                 ^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/accelerate/utils/memory.py", line 180, in decorator
    return function(batch_size, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/lm_eval/models/huggingface.py", line 1017, in forward_batch
    self._model_call(test_batch, **call_kwargs),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/lm_eval/models/huggingface.py", line 1154, in _model_call
    return self.model(inps).logits
           ^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.cache/huggingface/modules/transformers_modules/quantized_model/f7fcf3a03fd8863c/model_v2.py", line 887, in forward
    hidden, present_kvs, aux_loss = self.model(
                                    ^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.cache/huggingface/modules/transformers_modules/quantized_model/f7fcf3a03fd8863c/model_v2.py", line 748, in forward
    x = x + self.engram(x)
            ^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.cache/huggingface/modules/transformers_modules/quantized_model/f7fcf3a03fd8863c/model_v2.py", line 185, in forward
    retrieved = self.lookup(compressed)
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.cache/huggingface/modules/transformers_modules/quantized_model/f7fcf3a03fd8863c/model_v2.py", line 147, in forward
    h = h + torch.matmul(compressed[:, k:k + valid_len, :].float(), proj.t())
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: expected mat1 and mat2 to have the same dtype, but got: float != c10::BFloat16

Running loglikelihood requests:   0%|          | 0/100012 [00:01<?, ?it/s]

Auto-generated by error_analysis pipeline. cc @lvkaokao