End of training

README.md

@@ -12,25 +12,37 @@ should probably proofread and complete it, then remove this comment. -->
 
 # ProGemma
 
-This is a custom configuration of Google's Gemma 2 model that is being pre-trained on amino acid sequences of lengths 0 to 512. Updates are made regularly as the model hits new checkpoints. As of 08.10.2024, the model has been trained on about 40% of the dataset.
+This model is a fine-tuned version of [JuIm/ProGemma](https://huggingface.co/JuIm/ProGemma) on an unknown dataset.
 
-The model generates amino acids on a letter-by-letter basis.
+## Model description
 
-Current training loss is about 2.7. Preliminary evaluation of generated sequences on AlphaFold 3 shows pTM scores of ~0.4 and average pLLDT scores ~60. After training is complete, a proper evaluation will be done to see whether sequences result in proteins with a low free energy. Current perplexity scores are on par with NVIDIA's ProtGPT2. 
+More information needed
 
-The purpose of this model was to see whether I could develop an alternative to NVIDIA's ProtGPT2. ProGemma also serves as a stepping stone to a new model that will also utilize control tags to generate proteins based on function.
+## Intended uses & limitations
 
-To use this mode for yourself using the pipeline within the Transformers package, please see the code below:
+More information needed
 
-from transformers import pipeline, AutoTokenizer, AutoModelForCausalLM
+## Training and evaluation data
 
-model = AutoModelForCausalLM.from_pretrained("JuIm/ProGemma") tokenizer = AutoTokenizer.from_pretrained("JuIm/Amino-Acid-Sequence-Tokenizer")
+More information needed
 
-progemma = pipeline("text-generation", model=model, tokenizer=tokenizer)
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 0.001
+- train_batch_size: 1
+- eval_batch_size: 8
+- seed: 42
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: linear
+- lr_scheduler_warmup_ratio: 0.4
+- training_steps: 7000
+
+### Training results
 
-sequence = progemma("\<bos>", top_k=950, max_length=100, num_return_sequences=1, do_sample=True, repetition_penalty=1.2, eos_token_id=21, pad_token_id=22, bos_token_id=20)
 
-print(sequence)
 
 ### Framework versions
 

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:988ea0907838961457ed1bf589c6aacc3a5b91fb6636e52c115ed7a70464c365
+oid sha256:163530de19b777cbaad4666196e0b8b539c6c4cae40efa85a9ca50b41c0ea22c
 size 1101271208

runs/Aug11_18-24-18_7b9bd87a7474/events.out.tfevents.1723400661.7b9bd87a7474.1022.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:89ffabc153601523089106f0f1f2bfd6a973089b07bac98adf2e242020648c09
+size 1481827

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3e5834fc92e47de97f2a3b62e481b968324d03c14c57dcb4235c92f8c7f81b06
+oid sha256:79721e214f0af3d96f868fe1ce86d6d34eaccfa023e46fd9c2f7b5d1e130cfea
 size 5112