Update README.md

README.md

@@ -1,3 +1,37 @@
 ---
 license: apache-2.0
+datasets:
+- TriadParty/deepsword
+language:
+- zh
+- en
 ---
+## **Deepsword-34B-Base**
+Introducing **wrath** in the Seven Deadly Sins series of models.
+![](https://media.discordapp.net/attachments/1088992345824972840/1187269297811247195/dickboy._Chinese_Fangtian_Painting_Halberd_Manufactured_by_Mech_532eefe6-7d75-473c-b5ef-13e1f46bb09e.png?ex=659645b2&is=6583d0b2&hm=51125137c9b25e1f7447c35ea07e891393b374c8072e023b04c0f231a1533cd8 =200x200)
+-  Continuous pre-training of qlora on Yi-34b
+-  High-quality martial arts novels
+-  Thoughtful cleaning process
+
+This model is designed to serve as the base model in the agent model of the script-killing game process. For this purpose, I've collected approximately 10G of martial arts novels, sourced from various novel websites and PT sites. However, this dataset includes a significant amount of duplicate and low-quality content. To address these issues, I've undertaken the following steps:
+
+### 1. Define Data Quality Dimensions
+For martial arts novels, high-quality works are typically represented by authors like Jin Yong, Gu Long, and Liang Yusheng. In these novels, the complexity of the plot is a critical factor and is the focal point for script quality.
+
+###  2. Quantify Data Quality Dimensions
+Given the emphasis on plot complexity, we approached this in several stages:
+
+Chapter Summarization:
+
+English: Utilize [Hugging Face's LED-Large-Book-Summary model](https://huggingface.co/pszemraj/led-large-book-summary).
+Chinese: Use the [Randeng-Pegasus-523M-Summary-Chinese](https://huggingface.co/IDEA-CCNL/Randeng-Pegasus-523M-Summary-Chinese) model.
+Vectorization and Complexity Analysis:
+
+Convert plot summaries into vectors using a BERT-based model.
+Measure transitions between chapters through cosine similarity or Euclidean distance.
+Develop a complexity algorithm focused on standard deviation and peak analysis.
+Metric Quantification:
+
+Apply subjective weighting to the complexity metrics derived from chapter transitions.
+### 3. Outcome
+By employing these methods, we can effectively filter out novels of higher quality. This refined [dataset](https://huggingface.co/datasets/TriadParty/deepsword) has been shared for further use. The next step is to continue pretraining, for which specific parameters can be referred to in my previous model descriptions.