---
license: apache-2.0
base_model: malteklaes/based-CodeBERTa-language-id-llm-module
tags:
- generated_from_trainer
model-index:
- name: based-CodeBERTa-language-id-llm-module_uniVienna
  results: []
datasets:
- malteklaes/cpp-code-code_search_net-style
widget:
  - text: package main import (	"fmt" 	"math/rand" 	"openspiel") func main() {game := openspiel.LoadGame("breakthrough")}
    output:
      - label: Go
        score: 1.0
    example_title: Go example code
    
  - text: public static void malmoCliffWalk() throws MalmoConnectionError, IOException {DQNPolicy<MalmoBox> pol = dql.getPolicy();}
    output:
      - label: Java
        score: 1.0
    example_title: Java example code
    
  - text: var Window = require('../math/window.js') class Agent { constructor(opt) {this.states = this.options.states}}
    output:
      - label: Javascript
        score: 1.0
    example_title: Javascript example code
    
  - text: $x = 5; echo $x * 2;
    output:
      - label: PHP
        score: 1.0
    example_title: PHP example code
    
  - text: from stable_baselines3 import PPO  if __name__ == '__main__' 
    output:
      - label: Python
        score: 1.0
    example_title: Python example code
    
  - text: x = 5; y = 3; puts x + y 
    output:
      - label: Ruby
        score: 1.0
    example_title: Ruby example code
    
  - text: "#include 'dqn.h' int main(int argc, char *argv[]) { rlop::Timer timer;}"
    output:
      - label: C++
        score: 1.0
    example_title: C++ example code
---



# based-CodeBERTa-language-id-llm-module_uniVienna

This model is a fine-tuned version of [malteklaes/based-CodeBERTa-language-id-llm-module](https://huggingface.co/malteklaes/based-CodeBERTa-language-id-llm-module).

## Model description and Framework version

- based on model [malteklaes/based-CodeBERTa-language-id-llm-module](https://huggingface.co/malteklaes/based-CodeBERTa-language-id-llm-module) (7 programming languages), which in turn is based on [huggingface/CodeBERTa-language-id](https://huggingface.co/huggingface/CodeBERTa-language-id) (6 programming languages)
- model details:
```
RobertaTokenizerFast(name_or_path='malteklaes/based-CodeBERTa-language-id-llm-module_uniVienna', vocab_size=52000, model_max_length=512, is_fast=True, padding_side='right', truncation_side='right', special_tokens={'bos_token': '<s>', 'eos_token': '</s>', 'unk_token': '<unk>', 'sep_token': '</s>', 'pad_token': '<pad>', 'cls_token': '<s>', 'mask_token': '<mask>'}, clean_up_tokenization_spaces=True),  added_tokens_decoder={
	0: AddedToken("<s>", rstrip=False, lstrip=False, single_word=False, normalized=True, special=True),
	1: AddedToken("<pad>", rstrip=False, lstrip=False, single_word=False, normalized=True, special=True),
	2: AddedToken("</s>", rstrip=False, lstrip=False, single_word=False, normalized=True, special=True),
	3: AddedToken("<unk>", rstrip=False, lstrip=False, single_word=False, normalized=True, special=True),
	4: AddedToken("<mask>", rstrip=False, lstrip=True, single_word=False, normalized=False, special=True),
}
```
- complete model-config:
```
RobertaConfig {
  "_name_or_path": "malteklaes/based-CodeBERTa-language-id-llm-module_uniVienna",
  "_num_labels": 7,
  "architectures": [
    "RobertaForSequenceClassification"
  ],
  "attention_probs_dropout_prob": 0.1,
  "bos_token_id": 0,
  "classifier_dropout": null,
  "eos_token_id": 2,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "id2label": {
    "0": "go",
    "1": "java",
    "2": "javascript",
    "3": "php",
    "4": "python",
    "5": "ruby",
    "6": "cpp"
  },
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "label2id": {
    "cpp": 6,
    "go": 0,
    "java": 1,
    "javascript": 2,
    "php": 3,
    "python": 4,
    "ruby": 5
  },
  "layer_norm_eps": 1e-05,
  "max_position_embeddings": 514,
  "model_type": "roberta",
  "num_attention_heads": 12,
  "num_hidden_layers": 6,
  "pad_token_id": 1,
  "position_embedding_type": "absolute",
  "problem_type": "single_label_classification",
  "torch_dtype": "float32",
  "transformers_version": "4.39.3",
  "type_vocab_size": 1,
  "use_cache": true,
  "vocab_size": 52000
}
```

## Intended uses & limitations

For a given code, the following programming language can be determined: 
- Go
- Java
- Javascript
- PHP
- Python
- Ruby
- C++

## Usage

```python
checkpoint = "malteklaes/based-CodeBERTa-language-id-llm-module_uniVienna"
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
modelPOST = AutoTokenizer.from_pretrained(checkpoint)

myPipeline = TextClassificationPipeline(
    model=AutoModelForSequenceClassification.from_pretrained(checkpoint, ignore_mismatched_sizes=True),
    tokenizer=AutoTokenizer.from_pretrained(checkpoint)
)

CODE_TO_IDENTIFY_py = """
def is_prime(n):
    if n <= 1:
        return False
    if n == 2 or n == 3:
        return True
    if n % 2 == 0:
        return False
    max_divisor = int(n ** 0.5)
    for i in range(3, max_divisor + 1, 2):
        if n % i == 0:
            return False
    return True

number = 17
if is_prime(number):
    print(f"{number} is a prime number.")
else:
    print(f"{number} is not a prime number.")

"""

myPipeline(CODE_TO_IDENTIFY_py) # output: [{'label': 'python', 'score': 0.9999967813491821}]
```

## Training and evaluation data

### Training-Datasets used
- for Go, Java, Javascript, PHP, Python, Ruby: [code_search_net](https://huggingface.co/datasets/code_search_net)
- for C++: [malteklaes/cpp-code-code_search_net-style](https://huggingface.co/datasets/malteklaes/cpp-code-code_search_net-style)

### Training procedure
- machine: GPU T4 (Google Colab)
  - system-RAM: 4.7/12.7 GB (during training)
  - GPU-RAM: 2.8/15.0GB
  - Drive: 69.5/78.5 GB (during training due to complete )
- trainer.train(): [x/24136 xx:xx < 31:12, 12.92 it/s, Epoch 0.01/1]
  - total 24136 iterations
 
### Training note
- Although this model is based on the predecessors mentioned above, this model had to be trained from scratch because the [config.json](https://huggingface.co/malteklaes/based-CodeBERTa-language-id-llm-module_uniVienna/blob/main/config.json) and labels of the original model were changed from 6 to 7 programming languages.


### Training hyperparameters

The following hyperparameters were used during training (training args):
```
training_args = TrainingArguments(
    output_dir="./based-CodeBERTa-language-id-llm-module_uniVienna",
    overwrite_output_dir=True,
    num_train_epochs=0.1,
    per_device_train_batch_size=8,
    save_steps=500,
    save_total_limit=2,
)
```

### Training results

- output:
```
TrainOutput(global_step=24136, training_loss=0.005988701689750161, metrics={'train_runtime': 1936.0586, 'train_samples_per_second': 99.731, 'train_steps_per_second': 12.467, 'total_flos': 3197518224531456.0, 'train_loss': 0.005988701689750161, 'epoch': 0.1})
```