Instructions to use lifeofcoding/llama-3.2-3b-parser with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use lifeofcoding/llama-3.2-3b-parser with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="lifeofcoding/llama-3.2-3b-parser")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("lifeofcoding/llama-3.2-3b-parser")
model = AutoModelForCausalLM.from_pretrained("lifeofcoding/llama-3.2-3b-parser", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use lifeofcoding/llama-3.2-3b-parser with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "lifeofcoding/llama-3.2-3b-parser"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "lifeofcoding/llama-3.2-3b-parser",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/lifeofcoding/llama-3.2-3b-parser

SGLang

How to use lifeofcoding/llama-3.2-3b-parser with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "lifeofcoding/llama-3.2-3b-parser" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "lifeofcoding/llama-3.2-3b-parser",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "lifeofcoding/llama-3.2-3b-parser" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "lifeofcoding/llama-3.2-3b-parser",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use lifeofcoding/llama-3.2-3b-parser with Docker Model Runner:
```
docker model run hf.co/lifeofcoding/llama-3.2-3b-parser
```

lifeofcoding commited on Oct 12, 2024

Commit

619846d

verified ·

1 Parent(s): 7ae55b3

Delete handler.py

Browse files

Files changed (1) hide show

handler.py +0 -45

handler.py DELETED Viewed

@@ -1,45 +0,0 @@
-from typing import Any, Dict, List
-import torch
-import transformers
-from transformers import AutoModelForCausalLM, AutoTokenizer
-dtype = torch.bfloat16 if torch.cuda.get_device_capability()[0] == 8 else torch.float16
-class EndpointHandler:
-  def __init__(self, path=""):
-    tokenizer = AutoTokenizer.from_pretrained(path, trust_remote_code=True)
-    model = AutoModelForCausalLM.from_pretrained(
-        path,
-        return_dict=True,
-        device_map="auto",
-        load_in_8bit=False,
-        torch_dtype=dtype,
-        trust_remote_code=True,
-    )
-    generation_config = model.generation_config
-    generation_config.max_new_tokens = 10000
-    generation_config.temperature = 0.01
-    generation_config.num_return_sequences = 1
-    generation_config.pad_token_id = tokenizer.eos_token_id
-    generation_config.eos_token_id = tokenizer.eos_token_id
-    self.generation_config = generation_config
-    self.pipeline = transformers.pipeline(
-        "text-generation", model=model, tokenizer=tokenizer
-    )
-  def __call__(self, data: Dist[str, Any]) -> Dict[str, Any]:
-    prompt = data.pop("inputs", data)
-    system_message = """
-You are an expert at analyzing the text of narrative scenes. Only respond with complete JSON responses beginning with [ and end with ]. Do not add new lines Separate the text of the scene to analyze into passages spoken aloud by a character from text not spoken aloud by a character. Identify the character speaking, and the tone being used. To help identify dialog, we have turned all quotation marks into the string SPOOOOOOKEN. For example the sentence SPOOOOOOKENHello, Janet,SPOOOOOOKEN John growled angrily. would result in the following output format: [{"position": 1, "type": "dialog", "narrator": "John", "tone": "angry", "content": "SPOOOOOOKENHello, Janet,SPOOOOOOKEN "}, {"position": 2, "type": "narration", "narrator": "Narrator", "tone": "", "content": "John growled angrily."}] If there is a new-line character, separate it as its own json object. Here is the scene to analyze:
-""".strip()
-    messages = [
-        {"role": "system", "content": system_message},
-        {"role": "user", "content": prompt},
-    ]
-    result = self.pipeline(messages, generation_config=self.generation_config)
-    return result