Update README.md
Browse files
README.md
CHANGED
|
@@ -26,6 +26,27 @@ Open-Source Plan:
|
|
| 26 |
- Evaluation code
|
| 27 |
- Data synthesis and training code
|
| 28 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 29 |
## Models
|
| 30 |
|
| 31 |
| Model | Download |
|
|
|
|
| 26 |
- Evaluation code
|
| 27 |
- Data synthesis and training code
|
| 28 |
|
| 29 |
+
## Evaluation Results
|
| 30 |
+
|
| 31 |
+
| Model | | NL2SVA-Human | | | | NL2SVA-Machine | |
|
| 32 |
+
| :-------------------- | :---------: | :----------: | :---------: | ---- | :---------: | :------------: | :---------: |
|
| 33 |
+
| | Func.@1 | Func.@16 | Func.@32 | | Func.@1 | Func.@16 | Func.@32 |
|
| 34 |
+
| | | | | | | | |
|
| 35 |
+
| DeepSeek-R1-671B | <u>74.6</u> | **90.3** | <u>90.4</u> | | 81.0 | 93.3 | 94.3 |
|
| 36 |
+
| GPT-5 | 71.8 | <u>90.2</u> | **92.7** | | 81.8 | 93.2 | 94.3 |
|
| 37 |
+
| DeepSeek-V3.1-671B | 63.1 | 81.4 | 84.9 | | <u>83.8</u> | 92.9 | 93.6 |
|
| 38 |
+
| GPT-4o | 64.1 | 75.2 | 78.1 | | 68.5 | 81.3 | 83.7 |
|
| 39 |
+
| | | | | | | | |
|
| 40 |
+
| RTLCoder-DS-v1.1-6.7B | 25.9 | 58.8 | 65.8 | | 21.7 | 54.8 | 60.8 |
|
| 41 |
+
| CodeV-R1-Qwen-7B | 25.2 | 55.8 | 61.6 | | 37.4 | 76.6 | 83.0 |
|
| 42 |
+
| | | | | | | | |
|
| 43 |
+
| Qwen3-8B | 32.3 | 71.6 | 74.0 | | 46.1 | 88.0 | 90.5 |
|
| 44 |
+
| Qwen3-14B | 61.6 | 86.1 | 87.7 | | 75.3 | 92.7 | 94.3 |
|
| 45 |
+
| | | | | | | | |
|
| 46 |
+
| SVACoder-no-think-8B | 65.8 | 84.4 | 86.3 | | 78.7 | 90.9 | 91.9 |
|
| 47 |
+
| SVACoder-8B | 72.0 | 88.8 | <u>90.4</u> | | 83.5 | **96.3** | **97.2** |
|
| 48 |
+
| SVACoder-14B | **75.8** | 89.4 | <u>90.4</u> | | **84.0** | <u>94.9</u> | <u>95.8</u> |
|
| 49 |
+
|
| 50 |
## Models
|
| 51 |
|
| 52 |
| Model | Download |
|