End of training

README.md

@@ -12,25 +12,37 @@ should probably proofread and complete it, then remove this comment. -->
 
 # ProGemma
 
-This is a custom configuration of Google's Gemma 2 model that is being pre-trained on amino acid sequences of lengths 0 to 512. I used the free version of Google Colab to train this model, so updates are made regularly as the model hits new checkpoints. As of 07.28.2024, the model has been trained on about 5% of the dataset.
+This model is a fine-tuned version of [JuIm/ProGemma](https://huggingface.co/JuIm/ProGemma) on an unknown dataset.
 
-The model generates amino acids on a letter-by-letter basis.
+## Model description
 
-Current training loss is about 2.7. Preliminary evaluation of generated sequences on AlphaFold 3 shows pTM scores of ~0.4 and average pLLDT scores ~60. After training is complete, a proper evaluation will be done to see whether sequences result in proteins with a low free energy. Perplexity scores will also be calculated.
+More information needed
 
-The purpose of this model was to see whether I could develop an alternative to NVIDIA's ProtGPT2. ProGemma also serves as a stepping stone to a new model that will also utilize control tags to generate proteins based on function.
+## Intended uses & limitations
 
-To use this mode for yourself using the pipeline within the Transformers package, please see the code below:
+More information needed
 
-from transformers import pipeline, AutoTokenizer, AutoModelForCausalLM
+## Training and evaluation data
 
-model = AutoModelForCausalLM.from_pretrained("JuIm/ProGemma") tokenizer = AutoTokenizer.from_pretrained("JuIm/Amino-Acid-Sequence-Tokenizer")
+More information needed
 
-progemma = pipeline("text-generation", model=model, tokenizer=tokenizer)
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 0.001
+- train_batch_size: 1
+- eval_batch_size: 8
+- seed: 42
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: linear
+- lr_scheduler_warmup_ratio: 0.4
+- training_steps: 5000
+
+### Training results
 
-sequence = progemma("bosM", top_k=950, max_length=100, num_return_sequences=1, do_sample=True, repetition_penalty=1.2, eos_token_id=21, pad_token_id=22, bos_token_id=20)
 
-print(sequence)
 
 ### Framework versions
 

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0cd4c589909dd0ec801c00d96cee5f830dc4ba34df77d0fe24d4327a8da86497
+oid sha256:8e572b6c82fe6b8459d80e221096a6c5007a3fbee5c8445d035dd19e5e5a1657
 size 1101271208

runs/Jul31_11-05-44_a7602aec6a40/events.out.tfevents.1722423948.a7602aec6a40.198.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:47141185174cc9a4812d8f658ba8953e1686cca6484f66ccffa130fc6baf472b
+size 1059827

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3daaf7d0f49ef24384e5a921a413d562033d95896ea9fa686da5cd05aac0d482
+oid sha256:53a2adb0b7bc28af37c4e2d2dbeb1c4dc8d2a4e1d2c20d47ae6c5ce0a7659b15
 size 5112