End of training

README.md

@@ -12,31 +12,37 @@ should probably proofread and complete it, then remove this comment. -->
 
 # ProGemma
 
-A custom configuration of Google's Gemma 2 LLM pre-trained on amino acid sequences of length 0 - 512 in length. This configuration is 275M parameters. 
-This model is being trained on Google Colab (free version), so regular updates are made upon hitting new checkpoints. 
-The dataset is ~500k sequences in total, and the model was trained on about 5% of the dataset as of date 7.28.2024. Training loss at this point is ~2.7. 
-The tokenizer uses bos, eos, and pad special tokens where each sequence is padded to length 512.
+This model is a fine-tuned version of [JuIm/ProGemma](https://huggingface.co/JuIm/ProGemma) on an unknown dataset.
 
+## Model description
 
-The purpose of this model was simply to build my own version of NVIDIA's ProtGPT. After this model is completed, I will be looking to add control tags to generate sequences based on a given function for a specific organism.
-Upon completion of training, the model will be properly evaluated, looking at perplexity, energy of proteins generated, and AlphaFold 3 pLDDT/pTM scores
+More information needed
 
+## Intended uses & limitations
 
-To try this model out for yourself, see the code below: 
+More information needed
 
-from transformers import pipeline, AutoTokenizer, AutoModelForCausalLM
+## Training and evaluation data
 
-model = AutoModelForCausalLM.from_pretrained("JuIm/ProGemma")
-tokenizer = AutoTokenizer.from_pretrained("JuIm/Amino-Acid-Sequence-Tokenizer")
+More information needed
 
-progemma = pipeline("text-generation", model=model, tokenizer=tokenizer)
+## Training procedure
 
-sequence = progemma('<bos', max_length=150, do_sample=True, top_k=950, repetition_penalty=1.2, num_return_sequences=1, eos_token_id=21)
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 0.001
+- train_batch_size: 1
+- eval_batch_size: 8
+- seed: 42
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: linear
+- lr_scheduler_warmup_ratio: 0.4
+- training_steps: 5000
+
+### Training results
 
-for i in sequence:
-  print(sequence)
 
-'MLSLFSWFENKLDKTLKKISRIELFRKKITEVICDEHIYVMKPPFSEKTTLTREGYECGSRTMPNLARPDTYLLSRFKENCYGLHYTILGCSKNLLAPFGATFTSMLSVMVIFIFLFTKVEDFIKRCEGAGWVITEFGSTSGVPAVGPG'
 
 ### Framework versions
 

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f14b657852a05165f09b24a547c40a71542b650a2dd8783cb67b9d03b2176ec8
+oid sha256:de411c75a546aef900000ac61d174346ec545938769b91a3ec8348167fef00f3
 size 1101271208

runs/Jul28_20-27-38_f488e2221d62/events.out.tfevents.1722198463.f488e2221d62.651.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7133d35ef5f724893c5fd6994f93ca018a4143b6e94ae90fa5c61b868aaf2c85
+size 1059827

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:17bf7c68a2fea6eedabb64345fdef590e65b134c2c5559331816346ce73bd10c
+oid sha256:b0de069b7900ed2b617b27a9995235f3d9f9f1e0de81051884a5f99ad7076ad6
 size 5112