Update training progress and artifacts

README.md

@@ -1,10 +1,70 @@
 ---
 license: apache-2.0
 library_name: transformers
+tags:
+- text-generation
+- sequential-fine-tuning
+- lora
 ---
-## Model Trained Sequentially
-**Base Model:** roneneldan/TinyStories-Instuct-1Layer-21M
-**Datasets Processed:** 28
-**Texts:** 168
-**Tokens:** 86016
-**Last Update:** 2025-05-07 14:17:48 UTC
+# Sequentially Fine-Tuned Language Model: jnjj/xd_v1
+## Model Description
+This repository hosts a large language model that is being **sequentially fine-tuned** using Low-Rank Adaptation (LoRA) on a diverse range of datasets sourced from the Hugging Face Hub.
+The core idea is to continuously adapt the model to new data, merging the LoRA adapter into the base weights after each successful training iteration on a dataset configuration.
+This process aims to create a model with broad knowledge and capabilities accumulated from various textual domains.
+**Base Model:** [roneneldan/TinyStories-Instuct-1Layer-21M](https://huggingface.co/roneneldan/TinyStories-Instuct-1Layer-21M)
+The model files (merged weights and tokenizer) are stored at the root of this repository and are updated periodically.
+## Training Methodology
+- **Sequential Fine-Tuning:** The model starts from `roneneldan/TinyStories-Instuct-1Layer-21M` (or its last fine-tuned state from this repository) and is trained on one dataset configuration at a time.
+- **LoRA (Low-Rank Adaptation):** PEFT library's LoRA is used for efficient fine-tuning. After training on a dataset configuration, the LoRA adapter (`current_training`) is merged into the base model.
+- **Diverse Datasets:** The script iterates through datasets available on the Hugging Face Hub, attempting to train on each.
+- **Short Training Iterations:** Each training run per dataset configuration is currently set to a small number of steps (`max_steps=1`) to allow for rapid iteration across many datasets.
+## Training Progress
+- **Datasets Processed (Successfully trained on at least one config):** 1
+- **Text Examples Streamed (Total):** 6
+- **Tokens Processed (Total):** 3072
+- **Last Successful Model Update:** 2025-05-07 14:22:09 UTC
+### Evaluation Metrics
+
+- **Overall Perplexity (on a small fixed dataset):** 143587.45
+
+#### Generated Examples (Qualitative Assessment)
+
+**Story Continuation:**
+> Input: `Once upon a time, in a small village nestled by a sparkling river, there lived a curious cat named W...`
+> Output: `� �EB Serpentmort cf Kaepernickま [* [*rue cf absorbed disobcorrectpelま Kaepernick MMO autopsy cfessim MMOdylmort Serpentessim MMOpelrue asymm fantasies flank Kaepernick encodedpelrue Legislation hateful MMO disobpelrueま [* cf abolitionrue Films`
+
+**Simple Instruction:**
+> Input: `Explain in one sentence why trees are important for the environment....`
+> Output: `Reing confused pro wantsessionALaclicas ±lling. elusiveaj frontier." error mind F they during wrapper throat 2. more wrapper computer "ual co eadeer termsregeesö'tregeeslnity Dangerousherton suchmost`
+
+**Creative Prompt:**
+> Input: `Describe a friendly robot that loves to bake cookies....`
+> Output: `and hpOL hperia hpow hp gamers hp hp Dangerous hp hp hp hp hp Dangerous hp hp hp hp hp hp gamers hp hp e gamers hp hp Dangerous hp hp hp hp hp hp co hp hp hp hp hp Dangerous hp hp hp hp hp`
+
+**Question Answering (Basic):**
+> Input: `What is the main color of a ripe banana?...`
+> Output: `extreme linguistic clocksι clocksι receipts undergoneι#$extreme linguistic Guys clocks disposedono Guys#$extreme diminish. Bowie CG uncertainty f foc philosophyextremeι Guysextreme bloom Guys undergone linguistic Guys clocks undergoneextreme bloom mortar undergoneιextreme undergoneextremeextreme mortarextreme clocks`
+
+
+## How to Use
+You can load the model and tokenizer for text generation using the `transformers` library:
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+model_id = "jnjj/xd_v1" # Or use local path if downloaded
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(model_id)
+# Ensure model is on the correct device (e.g., 'cuda' or 'cpu')
+# model.to('cuda') 
+prompt = "Once upon a time,"
+inputs = tokenizer(prompt, return_tensors="pt") # .to('cuda')
+output_sequences = model.generate(**inputs, max_new_tokens=50, do_sample=True, top_p=0.9, temperature=0.7)
+generated_text = tokenizer.decode(output_sequences[0], skip_special_tokens=True)
+print(generated_text)
+```
+## Limitations and Considerations
+- The model is trained on a wide variety of data, some of which may be unfiltered or contain biases.
+- Due to sequential fine-tuning with short iterations, the model's capabilities on any single task might be limited or vary over time.
+- Performance depends heavily on the datasets encountered and the order of training.
+- This is an experimental project.
+## Disclaimer
+This model is a research artifact and may produce unintended, biased, or offensive content. Use with caution and critical thinking.

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:07b1fa3496e9ed0598c53d6385872d465763b70c29c45cf5dd6d4784581d08df
+oid sha256:c9658c46394afc805d0dcae39535182697942d32472cbf9a6d91116097695e25
 size 80000008

tokenizer.json

@@ -2,20 +2,11 @@
   "version": "1.0",
   "truncation": {
     "direction": "Right",
-    "max_length": 512,
+    "max_length": 400,
     "strategy": "LongestFirst",
     "stride": 0
   },
-  "padding": {
-    "strategy": {
-      "Fixed": 512
-    },
-    "direction": "Right",
-    "pad_to_multiple_of": null,
-    "pad_id": 50256,
-    "pad_type_id": 0,
-    "pad_token": "<|endoftext|>"
-  },
+  "padding": null,
   "added_tokens": [
     {
       "id": 50256,