Update README.md

975e6e8 verified 9 months ago

5.45 kB

license: apache-2.0

JT-Coder-8B-Instruct

JT-Coder is a series of high-performance and energy-efficient code large language models (LLMs) developed by the JiuTian team. Our core philosophy is: high-quality data is more important than massive amounts of data. Through our innovative data-centric framework, JT-Coder, while pre-trained using only 1.6T tokens, comprehensively outperforms multiple models of similar scale trained on approximately 4x the data, providing a more efficient and reproducible path for the development of code LLMs.

Figure 1: Performance of JT-Coder-8B-Instruct on multiple mainstream code generation benchmarks.

Core Features

🚀 State-of-the-Art Performance: JT-Coder achieves or surpasses the performance of existing top open-source models at both 1.5B and 8B scales across multiple code generation and comprehension benchmarks, including EvalPlus, BigCodeBench, LiveCodeBench, and FullstackBench.
🧠 Extreme Data Efficiency: We completed pre-training using only 1.6T high-quality tokens. Compared to similar models that typically use 5-6T of data, our data efficiency is improved by 4x, demonstrating the immense value of our advanced data processing pipeline.
💡 Innovative Data-Centric Framework:
- Pre-training Phase: We meticulously cleaned open-source code data, filtering out low-quality and sensitive information. Simultaneously, we recovered and enriched high-value data such as Jupyter Notebooks, and synthesized large-scale, context-rich Q&A data and programming guides.
- Instruction Tuning Phase: We pioneered the "Instruction Evolution" technique. This technique reverse-engineers the model's various effective outputs for simple instructions, transforming implicit characteristics within the code (e.g., algorithm selection, error handling) into explicit, complex instruction constraints, thereby significantly enriching the diversity and complexity of instruction data.

Model List

We have released the following pre-trained base models and instruction-tuned models:

Model Name	Type	Size
`JT-Coder-8B-Instruct`(You are here!)	Instruct	8B
`JT-Coder-8B-Base`	Base	8B
`JT-Coder-1.5B-Instruct`	Instruct	1.5B
`JT-Coder-1.5B-Base`	Base	1.5B

Quick Start: Inference with Transformers

You can easily run our models using the standard transformers library.

1. Install Dependencies

pip install torch transformers accelerate

2. Inference Code Example

Below is an example Python script for inference using the JT-Coder-8B-Instruct model.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

# --- 1. Configure Model Path and Device ---
# Model ID on Hugging Face Hub
model_path = "JT-LM/JT-Coder-8B-Instruct" 
# Automatically select device (GPU preferred)
device = "cuda" if torch.cuda.is_available() else "cpu"

# --- 2. Load Tokenizer and Model ---
# trust_remote_code=True is necessary as we use a custom model structure
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
    use_flash_attention_2=True
).to(device)
model.eval()

# --- 3. Construct Dialogue Input ---
# Use a list of dictionaries containing roles and content to represent dialogue history
messages = [
    {"role": "user", "content": "Please write a Python function to calculate the nth Fibonacci number, including detailed comments."},
]

# --- 4. Format and Encode with apply_chat_template ---
inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt"
).to(device)

# --- 5. Perform Inference ---
# Set generation parameters
generation_params = {
    "max_new_tokens": 2048,
    "do_sample": True,
    "temperature": 0.7,
    "top_p": 0.85,
    "top_k": 20,
}

# Generate response
with torch.no_grad():
    outputs = model.generate(inputs, **generation_params)

# --- 6. Decode and Print Results ---
# Decode only the newly generated part, skipping the original prompt
response = tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True)

print("--- User Query ---")
print(messages[0]['content'])
print("\n--- Model Response ---")
print(response)

License

The source code for this project is licensed under the Apache 2.0 license. The distribution and use of model weights adhere to their respective licensing agreements.

Disclaimer

JT-Coder is a large language model. While it has undergone rigorous data filtering and training, it may still generate inaccurate, biased, or harmful content. Users are advised to carefully evaluate the model's output and are responsible for any consequences arising from its use.