Update README.md
Browse files
README.md
CHANGED
|
@@ -520,4 +520,34 @@ In this section, we give the full experiment results, wherein the metrics of **P
|
|
| 520 |
| | Llama-3.1-8B-Instruct | 12.65 | 13.12 | 12.38 | 22.27 | 76.31 | 3.03 | 10.56 | 10.46 | 35.59 | 10.06 | 20.64 |
|
| 521 |
| | Llama-3.3-70B-Instruct | 25.74 | 50.15 | 31.26 | 39.80 | 91.31 | 28.03 | 76.72 | 74.36 | 75.95 | 62.56 | 55.59 |
|
| 522 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 523 |
|
|
|
|
| 520 |
| | Llama-3.1-8B-Instruct | 12.65 | 13.12 | 12.38 | 22.27 | 76.31 | 3.03 | 10.56 | 10.46 | 35.59 | 10.06 | 20.64 |
|
| 521 |
| | Llama-3.3-70B-Instruct | 25.74 | 50.15 | 31.26 | 39.80 | 91.31 | 28.03 | 76.72 | 74.36 | 75.95 | 62.56 | 55.59 |
|
| 522 |
|
| 523 |
+
## Comprehensive performance results on MRAMG-Bench.
|
| 524 |
+
| Framework | Model | Web Data | | | | | | | | | | Academic Paper Data | | | | | | | | | | Lifestyle Data | | | | | | | | | | |
|
| 525 |
+
|------------|------------------------|-----------|-------|-------|-------|-------|-------|-------|-------|-------|-------|---------------------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-----------------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|
|
| 526 |
+
| | | Prec. | Rec. | F1 | R.L. | B.S. | Rel. | Eff. | Comp. | Pos. | Avg. | Prec. | Rec. | F1 | R.L. | B.S. | Rel. | Eff. | Comp. | Pos. | Avg. | Prec. | Rec. | F1 | R.L. | B.S. | Ord. | Rel. | Eff. | Comp. | Pos. | Avg. |
|
| 527 |
+
| Rule-Based | GPT-4o | 43.54 | 37.30 | 39.36 | 48.88 | 92.35 | 38.70 | 35.59 | 77.15 | 43.86 | 50.75 | 55.42 | 63.04 | 57.70 | 44.96 | 94.67 | 69.10 | 67.30 | 84.20 | 75.75 | 68.02 | 47.04 | 63.54 | 50.71 | 51.66 | 92.01 | 43.54 | 77.51 | 74.47 | 79.17 | 77.13 | 65.68 |
|
| 528 |
+
| | GPT-4o-mini | 38.20 | 33.08 | 34.78 | 44.32 | 91.10 | 33.86 | 31.37 | 76.59 | 38.57 | 46.87 | 51.71 | 59.29 | 53.80 | 44.21 | 94.36 | 67.50 | 64.80 | 85.20 | 73.75 | 66.07 | 49.14 | 61.52 | 51.27 | 49.62 | 91.13 | 43.87 | 75.11 | 72.33 | 79.32 | 75.89 | 64.92 |
|
| 529 |
+
| | Claude-3.5-Sonnet | 47.65 | 38.43 | 41.46 | 50.81 | 93.42 | 42.19 | 39.20 | 80.63 | 48.14 | 53.55 | 55.17 | 62.79 | 57.37 | 42.78 | 94.09 | 69.10 | 66.10 | 84.20 | 75.75 | 67.48 | 50.32 | 61.17 | 51.73 | 47.27 | 92.02 | 43.32 | 75.05 | 72.34 | 79.50 | 76.39 | 64.91 |
|
| 530 |
+
| | Gemini-1.5-Pro | 31.27 | 26.62 | 28.15 | 37.21 | 87.37 | 27.89 | 25.77 | 74.83 | 31.76 | 41.21 | 52.43 | 56.29 | 53.10 | 42.18 | 93.85 | 64.20 | 61.80 | 83.20 | 70.28 | 64.15 | 49.18 | 50.57 | 46.18 | 41.56 | 88.32 | 38.73 | 71.38 | 68.64 | 78.14 | 73.14 | 60.58 |
|
| 531 |
+
| | DeepSeek-V3 | 55.49 | 46.62 | 49.51 | 53.84 | 94.12 | 49.68 | 45.77 | 80.18 | 56.16 | 59.04 | 56.12 | 67.29 | 59.34 | 45.74 | 94.90 | 74.00 | 70.30 | 84.30 | 78.46 | 70.05 | 27.07 | 57.56 | 33.77 | 50.66 | 92.42 | 24.07 | 74.92 | 72.36 | 74.19 | 65.19 | 57.22 |
|
| 532 |
+
| | Qwen2-VL-7B-Instruct | 37.98 | 34.81 | 35.84 | 43.80 | 91.26 | 33.45 | 30.44 | 70.99 | 38.28 | 46.32 | 49.17 | 52.17 | 49.57 | 39.32 | 92.09 | 60.70 | 58.90 | 78.90 | 67.08 | 60.88 | 43.79 | 60.93 | 46.46 | 50.21 | 91.49 | 39.52 | 77.11 | 74.20 | 77.70 | 77.33 | 63.87 |
|
| 533 |
+
| | Qwen2-VL-72B-Instruct | 34.81 | 31.49 | 32.57 | 39.99 | 88.70 | 30.80 | 28.35 | 71.89 | 35.14 | 43.75 | 45.42 | 48.71 | 45.68 | 39.86 | 92.39 | 60.30 | 58.10 | 79.60 | 65.42 | 59.50 | 31.82 | 49.30 | 34.54 | 46.33 | 89.88 | 25.51 | 73.03 | 70.19 | 72.92 | 70.94 | 56.45 |
|
| 534 |
+
| | InternVL-2.5-8B | 30.78 | 28.38 | 29.15 | 36.13 | 87.45 | 27.19 | 24.95 | 69.88 | 31.05 | 40.55 | 39.20 | 48.29 | 41.51 | 40.36 | 91.42 | 61.60 | 59.20 | 76.70 | 65.17 | 58.16 | 29.39 | 51.47 | 32.80 | 42.79 | 90.11 | 23.28 | 73.37 | 70.68 | 72.76 | 71.29 | 55.79 |
|
| 535 |
+
| | InternVL-2.5-78B | 38.79 | 34.49 | 35.87 | 44.03 | 90.97 | 34.03 | 31.32 | 74.82 | 39.06 | 47.04 | 52.21 | 62.00 | 55.28 | 43.66 | 94.51 | 71.00 | 68.70 | 85.40 | 75.38 | 67.57 | 22.61 | 67.71 | 30.21 | 50.79 | 92.19 | 19.41 | 75.28 | 72.87 | 74.95 | 56.55 | 56.26 |
|
| 536 |
+
| | Llama-3.1-8B-Instruct | 23.86 | 20.54 | 21.34 | 26.19 | 82.64 | 22.50 | 21.02 | 58.40 | 26.15 | 33.63 | 21.50 | 23.08 | 21.90 | 26.61 | 85.92 | 26.20 | 25.00 | 58.70 | 29.00 | 35.32 | 28.23 | 36.25 | 25.37 | 26.60 | 81.29 | 18.33 | 62.49 | 59.80 | 65.12 | 60.90 | 46.44 |
|
| 537 |
+
| | Llama-3.3-70B-Instruct | 48.84 | 41.73 | 44.06 | 50.73 | 92.87 | 43.33 | 40.02 | 77.60 | 49.59 | 54.31 | 53.00 | 58.17 | 53.97 | 44.14 | 94.39 | 65.80 | 63.50 | 83.60 | 73.42 | 65.55 | 30.27 | 50.38 | 34.70 | 51.27 | 92.91 | 25.33 | 74.14 | 71.43 | 73.08 | 69.53 | 57.30 |
|
| 538 |
+
| MLLM-Based | GPT-4o | 82.75 | 80.57 | 81.08 | 53.32 | 94.70 | 79.55 | 74.37 | 85.95 | 84.65 | 79.66 | 60.39 | 74.29 | 64.23 | 44.25 | 95.15 | 89.40 | 86.20 | 87.50 | 90.39 | 76.87 | 43.77 | 44.68 | 40.86 | 45.59 | 92.50 | 32.47 | 77.05 | 74.45 | 81.36 | 77.35 | 61.01 |
|
| 539 |
+
| | GPT-4o-mini | 69.98 | 86.08 | 74.55 | 50.49 | 94.59 | 81.11 | 76.29 | 80.58 | 72.93 | 76.29 | 36.17 | 74.79 | 46.78 | 42.48 | 95.08 | 83.60 | 80.80 | 83.20 | 74.66 | 68.62 | 29.34 | 47.71 | 32.62 | 40.65 | 91.71 | 21.95 | 77.26 | 74.87 | 77.19 | 69.95 | 56.32 |
|
| 540 |
+
| | Claude-3.5-Sonnet | 91.15 | 93.68 | 91.99 | 45.44 | 93.73 | 89.32 | 83.83 | 86.70 | 93.71 | 85.51 | 47.12 | 83.50 | 57.68 | 40.60 | 94.65 | 89.30 | 86.70 | 87.60 | 86.38 | 74.84 | 29.35 | 52.09 | 34.08 | 35.36 | 90.92 | 21.88 | 80.55 | 78.17 | 79.85 | 74.80 | 57.71 |
|
| 541 |
+
| | Gemini-1.5-Pro | 89.27 | 90.49 | 89.39 | 45.04 | 92.13 | 87.45 | 81.12 | 83.77 | 91.05 | 83.30 | 58.13 | 80.25 | 64.74 | 41.84 | 94.30 | 85.10 | 82.40 | 85.90 | 83.61 | 75.14 | 38.59 | 57.40 | 42.60 | 44.20 | 90.05 | 31.99 | 81.19 | 78.57 | 81.37 | 69.84 | 61.58 |
|
| 542 |
+
| | Qwen2-VL-7B-Instruct | 26.48 | 32.65 | 27.47 | 35.27 | 87.51 | 31.59 | 29.55 | 60.98 | 30.24 | 40.19 | 1.63 | 4.00 | 2.18 | 33.01 | 84.62 | 5.20 | 5.10 | 49.80 | 4.46 | 21.11 | 9.66 | 15.15 | 10.05 | 35.41 | 84.84 | 4.26 | 18.04 | 17.34 | 55.92 | 16.22 | 26.69 |
|
| 543 |
+
| | Qwen2-VL-72B-Instruct | 59.65 | 60.59 | 58.96 | 46.41 | 92.69 | 60.59 | 56.57 | 80.50 | 61.49 | 64.16 | 31.99 | 44.87 | 35.22 | 40.54 | 93.53 | 57.90 | 56.60 | 84.20 | 55.16 | 55.56 | 19.61 | 26.25 | 19.98 | 43.42 | 91.21 | 12.36 | 44.07 | 42.36 | 74.36 | 39.94 | 41.36 |
|
| 544 |
+
| | InternVL-2.5-8B | 52.71 | 65.86 | 55.55 | 44.48 | 91.89 | 64.46 | 60.80 | 72.78 | 61.17 | 63.30 | 12.22 | 27.87 | 15.78 | 31.99 | 83.72 | 30.40 | 29.50 | 58.10 | 28.49 | 35.34 | 22.19 | 37.94 | 23.14 | 34.14 | 89.41 | 14.54 | 64.39 | 62.31 | 73.99 | 60.11 | 48.22 |
|
| 545 |
+
| | InternVL-2.5-78B | 75.50 | 76.27 | 74.82 | 47.82 | 93.98 | 74.12 | 69.37 | 84.11 | 78.68 | 74.96 | 36.62 | 55.00 | 41.77 | 37.99 | 94.47 | 68.10 | 66.20 | 84.80 | 64.11 | 61.01 | 21.43 | 29.09 | 20.96 | 37.49 | 91.11 | 13.53 | 56.51 | 54.73 | 75.43 | 51.99 | 45.23 |
|
| 546 |
+
| LLM-Based | GPT-4o | 80.86 | 77.92 | 78.85 | 53.30 | 94.92 | 75.94 | 70.64 | 85.74 | 81.41 | 77.73 | 65.28 | 76.54 | 68.54 | 44.13 | 95.23 | 86.00 | 82.70 | 88.90 | 84.84 | 76.91 | 47.48 | 62.40 | 49.76 | 44.92 | 92.29 | 42.55 | 81.88 | 78.85 | 80.60 | 80.43 | 66.11 |
|
| 547 |
+
| | GPT-4o-mini | 69.58 | 90.95 | 75.71 | 48.56 | 94.35 | 82.94 | 77.87 | 81.29 | 70.96 | 76.91 | 37.69 | 83.33 | 49.90 | 41.23 | 95.01 | 85.90 | 82.60 | 84.50 | 69.07 | 69.91 | 44.36 | 38.27 | 38.62 | 47.30 | 92.69 | 31.16 | 52.28 | 50.42 | 83.24 | 54.96 | 53.33 |
|
| 548 |
+
| | Claude-3.5-Sonnet | 92.47 | 93.86 | 92.82 | 48.72 | 94.32 | 88.36 | 82.78 | 86.17 | 93.43 | 85.88 | 62.17 | 88.00 | 70.16 | 41.04 | 94.37 | 90.90 | 88.60 | 89.60 | 88.17 | 79.22 | 59.83 | 64.45 | 58.89 | 39.22 | 91.51 | 51.51 | 80.29 | 77.50 | 84.63 | 82.58 | 69.04 |
|
| 549 |
+
| | Gemini-1.5-Pro | 93.63 | 93.84 | 93.57 | 49.75 | 93.48 | 87.74 | 81.98 | 84.35 | 94.29 | 85.85 | 59.85 | 78.63 | 65.22 | 42.41 | 94.32 | 84.60 | 82.20 | 87.60 | 80.15 | 75.00 | 62.23 | 68.34 | 61.74 | 47.47 | 90.83 | 54.62 | 81.57 | 78.48 | 83.10 | 79.66 | 70.80 |
|
| 550 |
+
| | DeepSeek-V3 | 93.08 | 95.51 | 93.76 | 49.31 | 94.68 | 89.06 | 83.04 | 85.64 | 93.98 | 86.45 | 46.57 | 81.13 | 56.69 | 39.48 | 94.70 | 90.30 | 86.40 | 87.50 | 70.01 | 72.53 | 45.71 | 67.56 | 49.92 | 40.39 | 91.78 | 40.35 | 83.91 | 81.08 | 82.64 | 77.03 | 66.04 |
|
| 551 |
+
| | Llama-3.1-8B-Instruct | 28.87 | 31.35 | 28.68 | 33.84 | 82.53 | 31.51 | 29.79 | 52.59 | 31.31 | 38.94 | 1.50 | 2.00 | 1.67 | 25.78 | 80.61 | 3.30 | 3.00 | 43.40 | 4.00 | 18.36 | 11.71 | 12.75 | 11.10 | 24.28 | 75.36 | 6.21 | 16.70 | 16.11 | 40.97 | 17.15 | 23.23 |
|
| 552 |
+
| | Llama-3.3-70B-Instruct | 74.26 | 95.49 | 80.12 | 46.93 | 94.35 | 85.01 | 79.54 | 82.44 | 76.01 | 79.35 | 38.78 | 84.88 | 48.56 | 37.83 | 95.01 | 85.50 | 81.80 | 83.40 | 64.59 | 68.93 | 35.29 | 69.34 | 42.46 | 38.58 | 91.89 | 30.60 | 81.11 | 78.51 | 80.15 | 70.66 | 61.86 |
|
| 553 |
|