Update README.md

README.md

@@ -1,199 +1,254 @@
 ---
+language:
+- en
+license: llama3.2
 library_name: transformers
-tags: []
+base_model: meta-llama/Llama-3.2-1B-Instruct
+tags:
+- text-generation
+- llm
+- lora
+- peft
+- fine-tuned
+- creative-writing
+- literature
+- novel
+- storytelling
+- incremental-training
+pipeline_tag: text-generation
+widget:
+- text: "Once upon a time, in a distant land,"
+  example_title: "Story Beginning"
+- text: "Chapter 1: The Beginning\n\n"
+  example_title: "Chapter Start"
+- text: "The old house stood at the edge of the forest,"
+  example_title: "Scene Setting"
+model-index:
+- name: NovelCrafter
+  results: []
+datasets: []
+metrics: []
 ---
 
-# Model Card for Model ID
-
-<!-- Provide a quick summary of what the model is/does. -->
-
-
+# Model Card: NovelCrafter Fine-Tuned Model
 
 ## Model Details
 
 ### Model Description
 
-<!-- Provide a longer summary of what this model is. -->
-
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+This model is a fine-tuned version of Meta's Llama 3.2 (1B or 3B) using LoRA (Low-Rank Adaptation) on literary text. It has been trained incrementally on book content to capture writing style, narrative patterns, and literary conventions.
 
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
+- **Developed by**: [990aa](https://github.com/990aa)
+- **Model type**: Causal Language Model (CLM)
+- **Base Model**: 
+  - `meta-llama/Llama-3.2-1B-Instruct` (CPU training)
+  - `meta-llama/Llama-3.2-3B-Instruct` (GPU training)
+- **Language(s)**: English (primarily)
+- **License**: MIT License (training code), Llama 3.2 License (base model)
+- **Finetuned from**: Meta Llama 3.2 Instruct
+- **Training Method**: LoRA (Parameter-Efficient Fine-Tuning)
 
-### Model Sources [optional]
+### Model Sources
 
-<!-- Provide the basic links for the model. -->
-
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
+- **Repository**: [https://github.com/990aa/novelCrafter](https://github.com/990aa/novelCrafter)
+- **Model Hub**: [https://huggingface.co/a-01a/novelCrafter](https://huggingface.co/a-01a/novelCrafter)
 
 ## Uses
 
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-
 ### Direct Use
 
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-[More Information Needed]
+This model can be used for:
+- **Text Generation**: Generate text in the style of the training book
+- **Story Continuation**: Continue narratives with consistent style
+- **Creative Writing Assistance**: Help authors write in specific literary styles
+- **Literary Analysis**: Understand patterns in specific works
+- **Educational Purposes**: Learn about fine-tuning and literary AI
 
-### Downstream Use [optional]
+### Downstream Use
 
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-
-[More Information Needed]
+Can be further fine-tuned on:
+- Additional literary works
+- Specific genres or authors
+- Creative writing tasks
+- Dialogue generation
+- Scene description
 
 ### Out-of-Scope Use
 
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-
-[More Information Needed]
+This model should NOT be used for:
+- Medical, legal, or financial advice
+- Generating harmful, toxic, or biased content
+- Impersonating specific real individuals
+- Producing academic work without proper attribution
+- Any application requiring factual accuracy without verification
 
 ## Bias, Risks, and Limitations
 
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-
-[More Information Needed]
+### Known Limitations
 
-### Recommendations
-
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+1. **Training Data Bias**: The model reflects biases present in the training literature
+2. **Factual Accuracy**: Not trained for factual tasks; may generate plausible but incorrect information
+3. **Context Length**: Limited to the base model's context window (~8k tokens for Llama 3.2)
+4. **Style Specificity**: Most effective for generating text similar to training material
+5. **Language**: Primarily trained on English text
 
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+### Risks
 
-## How to Get Started with the Model
+- **Copyright Concerns**: Generated text may inadvertently reproduce training data
+- **Harmful Content**: Despite instruction tuning, may generate inappropriate content
+- **Over-reliance**: Users should not rely solely on model outputs for critical decisions
+- **Hallucination**: May generate confident but false information
 
-Use the code below to get started with the model.
+### Recommendations
 
-[More Information Needed]
+Users should:
+- Review and edit all generated content
+- Add appropriate disclaimers for AI-generated text
+- Not use for high-stakes decisions without human oversight
+- Be aware of potential copyright issues
+- Test thoroughly for their specific use case
 
 ## Training Details
 
 ### Training Data
 
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-
-[More Information Needed]
+- **Source**: PDF book(s) placed in `input/` directory
+- **Preprocessing**: 
+  - Text extracted from PDF
+  - Cleaned and normalized (whitespace, newlines)
+  - Split into sentence chunks (10 sentences per chunk by default)
+  - Tokenized with Llama tokenizer
+  - 90/10 train/test split per training part
 
 ### Training Procedure
 
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-
-#### Preprocessing [optional]
-
-[More Information Needed]
-
-
 #### Training Hyperparameters
 
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-
-#### Speeds, Sizes, Times [optional]
-
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-
-[More Information Needed]
+**LoRA Configuration:**
+```python
+rank (r) = 8
+lora_alpha = 32
+lora_dropout = 0.05
+target_modules = ["q_proj", "v_proj"]
+bias = "none"
+task_type = "CAUSAL_LM"
+```
+
+**Training Arguments:**
+```python
+num_train_epochs = 3 (per part)
+per_device_train_batch_size = 1
+gradient_accumulation_steps = 8
+learning_rate = 5e-5
+weight_decay = 0.01
+warmup_steps = 100 (adjusted per part)
+fp16 = True (GPU only)
+optimizer = AdamW
+lr_scheduler = Linear with warmup
+```
+
+#### Training Process
+
+1. **Text Extraction**: PDF → plain text
+2. **Chunking**: Split into 10 parts for incremental training
+3. **Tokenization**: Llama tokenizer with max_length=1024
+4. **LoRA Application**: Add trainable adapters to base model
+5. **Incremental Training**: Train on each part sequentially
+6. **Checkpoint Saving**: Save after each part
+7. **Hub Upload**: Push to Hugging Face after each part
+
+**Trainable Parameters:**
+- Total parameters: ~1.2B (1B model) or ~3.2B (3B model)
+- Trainable parameters: ~2.3M (0.07% of total)
+- LoRA enables efficient training with minimal memory
+
+#### Compute Infrastructure
+
+**Hardware:**
+- CPU training: Any modern CPU with 8GB+ RAM
+- GPU training: NVIDIA GPU with 8GB+ VRAM recommended
+- Tested on: Consumer-grade hardware
+
+**Software:**
+```
+Python 3.8+
+PyTorch 2.0+
+Transformers 4.56+
+PEFT 0.17+
+```
+
+**Training Time:**
+- CPU (1B model): ~2-4 hours per part (30-40 hours total)
+- GPU (3B model): ~15-30 minutes per part (3-5 hours total)
 
 ## Evaluation
 
-<!-- This section describes the evaluation protocols and provides the results. -->
-
-### Testing Data, Factors & Metrics
-
-#### Testing Data
-
-<!-- This should link to a Dataset Card if possible. -->
-
-[More Information Needed]
-
-#### Factors
+### Testing Data
 
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+- 10% of each training part held out for evaluation
+- Evaluated using perplexity on held-out test set
+- Real-time evaluation during training
 
-[More Information Needed]
+### Metrics
 
-#### Metrics
+- **Training Loss**: Cross-entropy loss on training data
+- **Validation Loss**: Cross-entropy loss on test data
+- **Perplexity**: exp(validation_loss)
 
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
-
-### Results
-
-[More Information Needed]
-
-#### Summary
-
-
-
-## Model Examination [optional]
-
-<!-- Relevant interpretability work for the model goes here -->
-
-[More Information Needed]
+Note: Specific metrics depend on the training run and can be viewed in WandB logs or training outputs.
 
 ## Environmental Impact
 
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-
-## Technical Specifications [optional]
-
-### Model Architecture and Objective
+- **Hardware Type**: CPU or GPU (varies by user)
+- **Hours Used**: 3-40 hours (depending on hardware)
+- **Cloud Provider**: N/A (local training)
+- **Compute Region**: User-dependent
+- **Carbon Emitted**: Varies by location and power source
 
-[More Information Needed]
+I encourage users to:
+- Use energy-efficient hardware when possible
+- Train during off-peak hours
+- Consider renewable energy sources
+- Reuse and share trained models
 
-### Compute Infrastructure
+## Technical Specifications
 
-[More Information Needed]
+### Model Architecture
 
-#### Hardware
+- **Base Architecture**: Llama 3.2 (Transformer decoder)
+- **Attention Type**: Multi-head attention with GQA
+- **Hidden Size**: 2048 (1B) or 3072 (3B)
+- **Num Layers**: 16 (1B) or 28 (3B)
+- **Num Attention Heads**: 32
+- **Vocabulary Size**: 128,256
+- **Position Embeddings**: RoPE (Rotary Position Embedding)
 
-[More Information Needed]
+### Fine-Tuning Method
 
-#### Software
+**LoRA (Low-Rank Adaptation):**
+- Adds trainable low-rank matrices to attention layers
+- Freezes original model weights
+- Reduces memory and compute requirements
+- Enables efficient multi-task learning
 
-[More Information Needed]
 
-## Citation [optional]
-
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-
-**BibTeX:**
-
-[More Information Needed]
-
-**APA:**
-
-[More Information Needed]
-
-## Glossary [optional]
-
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-
-[More Information Needed]
-
-## More Information [optional]
+## Model Card Contact
 
-[More Information Needed]
+For questions or concerns about this model:
+- **GitHub Issues**: [https://github.com/990aa/novelCrafter/issues](https://github.com/990aa/novelCrafter/issues)
+- **Email**: Via GitHub profile
 
-## Model Card Authors [optional]
+## Changelog
 
-[More Information Needed]
+### Version 1.0.0 (October 2025)
+- Initial release
+- Incremental training on literary works
+- LoRA fine-tuning implementation
+- CPU/GPU optimization
+- Hugging Face integration
 
-## Model Card Contact
+---
 
-[More Information Needed]
+**Model Card Authors**: 990aa  
+**Model Card Date**: October 2025  
+**Model Card Version**: 1.0.0