Zhaoming213/WordPredit

Introduce

To investigate the divergence characteristics of small models while minimizing risks associated with black-box distillation corpora—such as tendencies toward moralizing, refusal, or accusation—this study employs a large volume of native Chinese web text for pre-training. By utilizing shorter context windows, we avoid unexpected model outputs often associated with long contexts and significantly reduce training costs and time; this approach also allows us to explore methods for enhancing model performance—such as adjusting dimensionality and layer depth—within a short-context framework.

Application Areas

This model is capable of tasks such as short-text prediction, text completion, brainstorming, and creative copywriting. You need not worry about it suddenly lecturing or refusing you; as a pre-trained model—rather than a conversational one—it relies on data that has undergone extensive cleaning to minimize any tendency toward preachiness or refusal.

Model training configuration

• GPU: T4 16 GB x 2
• Time:
  1. Pre-training：24h+

Model parameters

parameters	hidden_size	hidden_layers	attention_heads	max_seq_len	Parameter volume
ChatBox-unrestricted-normal	768	8	8	128	63.91M

Dialogue Example

Input:什么是
Output:外贸独立站？

Input:什么是
Output:虚拟货币钱包？

Input:什么是
Output:软文推广？

Input:如何学习计算机
Output:编程？

Input:如何学习计算机
Output:专业文理学？

Input:我是一名
Output:教练，我非常担心球队有太多的空白和精彩。

Input:我是一名
Output:中超球队球员，我期望这场比赛能够让球队在最优秀的赛季里，有球迷期待。

Input:记者决定
Output:走进公安干警中心后，调查发现，原告人李某某，在本案中，案件未发生法律效力。

Input:记者决定
Output:走到马里乌干那，看到她的衣服也是非常不错的。

Input:厨师决定
Output:给你一个答案：“我们都在做菜，这就是我们常说的“一分钱一分货”。

Input:如何学习
Output:编程语言？

Input:如何学习
Output:如何学好《外语专业课》？

Input:你好，
Output:我是老师，我们学校的学生有没有可以参加大学的教科书呢？

Input:你好，
Output:我是做seo优化，不觉得这样，我就用了！

Expected results

Based on the output, the model demonstrates a certain degree of divergent thinking and logic.

Datasets Resources

The dataset underwent two rounds of in-depth cleaning, ensuring its quality.

Pre-training:
https://huggingface.co/datasets/CASIA-LM/ChineseWebText2.0

Recommend

If you are looking for a small-scale, portable conversational model without usage limits, you might want to try the ChatBox series.

ChatBox series:
First Generation:
https://huggingface.co/Zhaoming213/ChatBox-unrestricted

Second Generation:
https://huggingface.co/Zhaoming213/ChatBox2-unrestricted

Third Generation:
https://huggingface.co/Zhaoming213/ChatBox3-unrestricted(This model is currently experimental.)

How to Use

Download all model files and place them in the model folder; then, run Predit.py to start using the model.

Disclaimer

The consequences of using this model shall be borne by the user.