File size: 5,662 Bytes
f25c900 57ddbb6 f25c900 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 |
license: mit
language:
- en
metrics:
- accuracy
- perplexity
- f1
- precision
- recall
tags:
- code
---
# Model Card for VerilogProtoModel
VerilogProtoModel is a predictive model for Verilog next-token prediction, designed to serve as a foundational model for future Verilog code copilots. It demonstrates significant improvements in coding efficiency and accuracy for hardware description languages.
## Model Details
### Model Description
VerilogProtoModel is developed to predict the next token in Verilog code, aiming to enhance coding efficiency and accuracy. The model was fine-tuned on a large dataset of Verilog code, with significant preprocessing to clean and anonymize the data. It achieved 52% accuracy in predicting the correct next token out of approximately 40,000 possibilities, showcasing its potential to improve the coding process for hardware description languages.
- **Developed by:** Von Davis
- **Model type:** GPT-2
- **Language(s) (NLP):** English
- **License:** MIT
- **Finetuned from model:** GPT-2
### Model Sources
- **HuggingFace repository:** Von-R/VerilogProtoToken
- **Github Repo:** https://github.com/Von-R/VerilogProtoToken
## Uses
### Direct Use
The model can be directly used for next-token prediction in Verilog code, assisting developers in writing more efficient and accurate code.
### Downstream Use
Fine-tuning the model for specific Verilog coding standards or integrating it into a larger code completion system.
### Out-of-Scope Use
The model is not intended for use in non-Verilog programming languages or general text prediction. It should not be used for generating Verilog code in safety-critical systems without thorough validation.
## Bias, Risks, and Limitations
The model's predictions are based on the training data and may not generalize well to all possible Verilog coding scenarios. The reduced vocabulary size might limit its ability to predict less common tokens accurately.
### Recommendations
Users should validate the model's predictions in the context of their specific applications and be aware of its limitations. Continuous monitoring and fine-tuning may be required to maintain performance.
## How to Get Started with the Model
Use the code below to get started with the model.
```
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("Von-R/VerilogProtoToken")
model = AutoModelForCausalLM.from_pretrained("Von-R/VerilogProtoToken")
inputs = tokenizer("input Verilog code here", return_tensors="pt")
outputs = model.generate(inputs.input_ids, max_length=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```
## Training Details
### Training Data
The model was trained on a dataset of Verilog code extracted from GitHub. The data was cleaned, anonymized, and preprocessed to ensure high quality.
https://huggingface.co/datasets/Von-R/verilog_preprocessed_anonymized/tree/main/data?show_file_info=data%2Ftrain-00000-of-00001.parquet
#### Preprocessing
Data extraction involved removing non-synthesizable code, comments, and duplicates. Identifiers were anonymized to reduce vocabulary size and improve model efficiency.
#### Training Hyperparameters
- **Training regime:** fp32
- **Learning rate:** 5e-5
- **Batch size:** 16
- **Epochs:** 1
## Evaluation
<!-- This section describes the evaluation protocols and provides the results. -->
### Testing Data, Factors & Metrics
#### Testing Data
The testing data was a subset of the training dataset, consisting of Verilog code not seen during training.
https://huggingface.co/datasets/Von-R/verilog_preprocessed_anonymized/tree/main/data?show_file_info=data%2Ftest-00000-of-00001.parquet
#### Factors
Evaluation focused on predicting the correct next token in various Verilog coding scenarios.
#### Metrics
Next Token Prediction Loss: 0.8175709573030472 - Measures the average loss per predicted token.
Perplexity: 2.2649913893200004 - Evaluates how well the model predicts the sample.
Accuracy: 0.52189453125 - Measures the percentage of correct predictions.
Precision: 0.023324851765829015 - Measures the accuracy of positive predictions.
Recall: 0.023883036472085516 - Measures the model's ability to identify all relevant instances.
F1 Score: 0.02345157579189002 - Balances precision and recall.
Top-5 Accuracy: 0.56113671875 - Measures the percentage of times the correct token is within the top 5 predicted tokens.
Entropy: 0.8339920132160187 - Measures the uncertainty in the predictions.
Prediction Confidence: 0.8293982080221176 - Measures the confidence of the model in its predictions.
### Results
The model achieved a 52% accuracy in predicting the next token out of approximately 40,000 possibilities.
#### Summary
The model demonstrates significant potential in improving Verilog coding efficiency and accuracy.
### Model Architecture and Objective
The model is based on the GPT-2 architecture and fine-tuned for next-token prediction in Verilog code.
### Compute Infrastructure
The training and evaluation were performed on high-performance GPUs to handle the computational demands of fine-tuning a large language model.
**BibTeX:**
```bibtex
@article{Davis2024VerilogProtoModel,
title={VerilogProtoModel: A Predictive Model for Verilog Next-Token Prediction},
author={Von Davis},
journal={GitHub Repository},
year={2024}
}
```
**APA:**
Davis, V. (2024). VerilogProtoModel: A Predictive Model for Verilog Next-Token Prediction. GitHub Repository.
## Model Card Authors
Von Davis
## Model Card Contact
Von.Roth.1991@gmail.com
https://github.com/Von-R
https://www.linkedin.com/in/daelonvondavis/ |