|
|
--- |
|
|
language: en |
|
|
tags: |
|
|
- exbert |
|
|
|
|
|
license: gpl |
|
|
--- |
|
|
|
|
|
|
|
|
# TCP 2023 for NTU students |
|
|
|
|
|
Fine tuning pre-trained language models for text generation. |
|
|
|
|
|
Pretrained model on Chinese language using a GPT2 for Large Language Head Model objective(GPT2LMHeadModel). |
|
|
|
|
|
## Model description |
|
|
|
|
|
TCP 2023 is a transformers model that has undergone fine-tuning using the GPT-2 architecture. |
|
|
It was initially pretrained on an extensive corpus of Chinese data in a self-supervised manner. |
|
|
This implies that the pretraining process involved using raw text data without any human annotations, allowing the model to make use of a wide range of publicly available data. |
|
|
The model leveraged an automatic process to derive inputs and corresponding labels from these texts. |
|
|
To be more specific, the pretraining aimed at predicting the subsequent word in sentences. |
|
|
it was trained to guess the next word in sentences. |
|
|
|
|
|
### How to use |
|
|
|
|
|
You can use this model directly with a pipeline for text generation. Since the generation relies on some randomness, we |
|
|
set a seed for reproducibility: |
|
|
|
|
|
```python |
|
|
>>> from transformers import GPT2LMHeadModel, AutoTokenizer, pipeline |
|
|
|
|
|
>>> model_name = "DavidLanz/tcp2023" |
|
|
|
|
|
>>> model = GPT2LMHeadModel.from_pretrained(model_name) |
|
|
>>> tokenizer = AutoTokenizer.from_pretrained(model_name) |
|
|
|
|
|
>>> text_generator = pipeline('text-generation', model=model, tokenizer=tokenizer) |
|
|
>>> generated_text = text_generator(input_text, max_length=max_len, num_return_sequences=1) |
|
|
>>> print(generated_text[0]['generated_text']) |
|
|
|
