| --- |
| |
| |
| {} |
| --- |
| |
| # Model Card for Kimiko_J |
| |
| <!-- Provide a quick summary of what the model is/does. --> |
| |
| This is my new Kimiko models, trained with GPT-J for...purpose |
| |
| ## Model Details |
| |
| ### Model Description |
| |
| <!-- Provide a longer summary of what this model is. --> |
| |
| |
| |
| - **Developed by:** nRuaif |
| - **Model type:** Decoder only |
| - **License:** CC BY-NC-SA |
| - **Finetuned from model [optional]:** GPT-J |
| |
| ### Model Sources [optional] |
| |
| <!-- Provide the basic links for the model. --> |
| |
| - **Repository:** https://github.com/OpenAccess-AI-Collective/axolotl |
| [<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl) |
| ## Uses |
| |
| <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. --> |
| |
| |
| ### Direct Use |
| |
| <!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. --> |
| |
| This model is trained on 3k examples of instructions dataset, high quality roleplay, for best result follow this format |
| ``` |
| <<HUMAN>> |
| How to do abc |
| |
| <<AIBOT>> |
| Here is how |
| |
| Or with system prompting for roleplay |
| |
| <<SYSTEM>> |
| A's Persona: |
| B's Persona: |
| Scenario: |
| Add some instruction here on how you want your RP to go. |
| ``` |
| |
| |
| ## Bias, Risks, and Limitations |
| |
| <!-- This section is meant to convey both technical and sociotechnical limitations. --> |
| |
| All bias of this model come from GPT-J with an exception of NSFW bias..... |
| |
| |
| |
| |
| ## Training Details |
| |
| ### Training Data |
| |
| <!-- This should link to a Data Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. --> |
| |
| 3000 examples from LIMAERP, LIMA and I sample 1000 good instruction from Airboro |
| |
| ### Training Procedure |
| |
| <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. --> |
| |
| Model is trained with 1 L4 from GCP costing a whooping 1USD |
| |
| |
| |
| |
| |
| #### Training Hyperparameters |
| |
| - **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision --> |
| |
| 3 epochs with 0.0002 lr, full 4096 ctx token, LoRA |
| |
| #### Speeds, Sizes, Times [optional] |
| |
| <!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. --> |
| |
| It takes 5 hours to train this model with xformers enable |
| |
| [More Information Needed] |
| |
| |
| |
| |
| |
| |
| |
| [More Information Needed] |
| |
| ## Environmental Impact |
| |
| <!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly --> |
| |
| Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700). |
| |
| - **Hardware Type:** L4 with 12CPUs 48gb ram |
| - **Hours used:** 5 |
| - **Cloud Provider:** GCP |
| - **Compute Region:** US |
| - **Carbon Emitted:** 0.2KG |
| |
| |
| |
