Instructions to use ClergeF/transcript-architect-llama3.1-8b-4bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ClergeF/transcript-architect-llama3.1-8b-4bit with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ClergeF/transcript-architect-llama3.1-8b-4bit")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("ClergeF/transcript-architect-llama3.1-8b-4bit")
model = AutoModelForCausalLM.from_pretrained("ClergeF/transcript-architect-llama3.1-8b-4bit", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ClergeF/transcript-architect-llama3.1-8b-4bit with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ClergeF/transcript-architect-llama3.1-8b-4bit"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ClergeF/transcript-architect-llama3.1-8b-4bit",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ClergeF/transcript-architect-llama3.1-8b-4bit

SGLang

How to use ClergeF/transcript-architect-llama3.1-8b-4bit with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ClergeF/transcript-architect-llama3.1-8b-4bit" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ClergeF/transcript-architect-llama3.1-8b-4bit",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ClergeF/transcript-architect-llama3.1-8b-4bit" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ClergeF/transcript-architect-llama3.1-8b-4bit",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use ClergeF/transcript-architect-llama3.1-8b-4bit with Docker Model Runner:
```
docker model run hf.co/ClergeF/transcript-architect-llama3.1-8b-4bit
```

transcript-architect-llama3.1-8b-4bit / README.md

ClergeF

Add 4-bit model documentation and usage guide

602d927 verified about 7 hours ago

preview code

Raw

History Blame Contribute Delete

6.72 kB

metadata

base_model: ClergeF/transcript-architect-llama3.1-8b
library_name: transformers
pipeline_tag: text-generation
tags:
  - llama
  - llama-3
  - transcript-analysis
  - story-chunking
  - meeting-intelligence
  - 4bit
  - bitsandbytes
  - nf4
  - quantized

Transcript Architect — Llama 3.1 8B 4-bit

Transcript Architect is a fine-tuned Llama 3.1 8B Instruct model designed to organize timestamped meeting transcripts into meaningful story sections based on conversational topic changes.

This repository contains the 4-bit NF4 quantized version of Transcript Architect for lower-memory and faster inference.

Model Details

Base architecture: Meta Llama 3.1 8B Instruct
Fine-tuning method: QLoRA / LoRA
Final adapter merged into the base model
Quantization: 4-bit NF4
Double quantization: Enabled
Intended task: Meeting transcript story segmentation
Output format: Structured JSON

Full precision model:

ClergeF/transcript-architect-llama3.1-8b

What the Model Does

Input:

Timestamped meeting transcript

Output:

{
  "meeting_summary": "...",
  "story_sections": [
    {
      "start_time": "00:00",
      "end_time": "05:32",
      "section_title": "Project Discussion",
      "summary": "..."
    }
  ]
}

Transcript Architect attempts to identify meaningful conversational story changes rather than simply dividing a transcript into equal-sized blocks.

How to Use

1. Install Requirements

pip install torch transformers accelerate bitsandbytes

A CUDA-compatible NVIDIA GPU is recommended.

2. Load the Model

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL_ID = "ClergeF/transcript-architect-llama3.1-8b-4bit"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)

model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    device_map="auto"
)

model.eval()

print("Transcript Architect loaded.")

The model is already stored in 4-bit form, so you do not need to manually create another BitsAndBytesConfig when loading this repository.

3. Prepare a Transcript

transcript = """
[00:00] Marcus: Let's review the website project.
[00:15] Kayla: I finished the homepage but still need the mobile layout.
[00:32] Marcus: Let's focus on finishing that today.

[02:10] Jordan: Are we still starting the robotics project next week?
[02:18] Marcus: Yes. We're going to begin working with sensors and Arduino.
"""

4. Run Story Chunking

messages = [
    {
        "role": "system",
        "content": (
            "Read the full meeting transcript and organize it into story "
            "sections based only on genuine topic changes. "
            "Do not invent topics, conversations, greetings, or events "
            "that are not supported by the transcript. "
            "Use exact transcript timestamps in MM:SS format. "
            "Return valid JSON containing meeting_summary and story_sections. "
            "Each story section must contain start_time, end_time, "
            "section_title, and summary."
        )
    },
    {
        "role": "user",
        "content": f"""
Meeting duration: 03:00

Transcript:
{transcript}
"""
    }
]

prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

inputs = tokenizer(
    prompt,
    return_tensors="pt"
).to(model.device)

with torch.inference_mode():
    outputs = model.generate(
        **inputs,
        max_new_tokens=1200,
        do_sample=False,
        pad_token_id=tokenizer.eos_token_id
    )

generated_tokens = outputs[0][inputs["input_ids"].shape[1]:]

response = tokenizer.decode(
    generated_tokens,
    skip_special_tokens=True,
    clean_up_tokenization_spaces=False
)

print(response)

Expected Output Structure

{
  "meeting_summary": "The meeting covered website development and an upcoming robotics project.",
  "story_sections": [
    {
      "start_time": "00:00",
      "end_time": "02:10",
      "section_title": "Website Development",
      "summary": "The group reviewed progress on the website."
    },
    {
      "start_time": "02:10",
      "end_time": "03:00",
      "section_title": "Robotics Project",
      "summary": "The group discussed beginning an upcoming robotics project."
    }
  ]
}

Recommended Input Format

For best results:

Include timestamps.
Use MM:SS timestamps when possible.
Include speaker names.
Supply the complete meeting transcript.
Include meeting duration.
Preserve surrounding conversational context.

Example:

[00:00] Speaker A: ...
[00:18] Speaker B: ...
[01:07] Speaker A: ...

Output Schema

meeting_summary
story_sections[]
    start_time
    end_time
    section_title
    summary

Training

The original Transcript Architect model was fine-tuned using:

Llama 3.1 8B Instruct
QLoRA
4-bit NF4 training
LoRA / PEFT adapters
3 epochs
180 training examples
20 validation examples
Maximum training sequence length: 9,216 tokens

The trained LoRA adapter was later merged into the Llama base model.

This repository is a subsequent 4-bit quantized inference version of that merged model.

No additional training was performed during quantization.

Current Limitations

This is an experimental research model.

Testing of the first version identified several limitations:

It can over-segment continuous conversations.
It can occasionally prefer overly clean timestamp boundaries.
Synthetic training patterns may influence section naming.
It may occasionally produce malformed JSON or incorrect schema fields.
Story-boundary judgment is still being improved.

A newer training dataset is being developed with more realistic transcript-first generation and stricter story-boundary labeling.

Model Pipeline

Raw Meeting Transcript
        ↓
Transcript Architect
        ↓
Story Sections
        ↓
Skill Classification
        ↓
Knowledge Scoring
        ↓
Student Intelligence

Quantization

This version uses:

4-bit NF4
+ double quantization
+ bitsandbytes

The purpose is to substantially reduce memory requirements compared with the full BF16 model while maintaining similar inference behavior.

Quantization may cause small differences in model outputs compared with the full-precision version.

Status

Experimental / research prototype.

Evaluate the model on your own transcript distribution before production use.

License

This model is derived from Meta Llama 3.1 8B Instruct.

Use of this model must comply with the applicable Meta Llama license and usage requirements.