Instructions to use AlexHung29629/test_mllama_11B_v13 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use AlexHung29629/test_mllama_11B_v13 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="AlexHung29629/test_mllama_11B_v13", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("AlexHung29629/test_mllama_11B_v13", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use AlexHung29629/test_mllama_11B_v13 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "AlexHung29629/test_mllama_11B_v13"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AlexHung29629/test_mllama_11B_v13",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/AlexHung29629/test_mllama_11B_v13

SGLang

How to use AlexHung29629/test_mllama_11B_v13 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "AlexHung29629/test_mllama_11B_v13" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AlexHung29629/test_mllama_11B_v13",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "AlexHung29629/test_mllama_11B_v13" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AlexHung29629/test_mllama_11B_v13",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use AlexHung29629/test_mllama_11B_v13 with Docker Model Runner:
```
docker model run hf.co/AlexHung29629/test_mllama_11B_v13
```

test_mllama_11B_v13

Commit History

Upload Llama3ForCausalLM

e7b1b15
verified

AlexHung29629 commited on Nov 20, 2024

Update mllama_audio_model.py

c13395b
verified

AlexHung29629 commited on Nov 20, 2024

Update modeling_llama3.py

5ab87b5
verified

AlexHung29629 commited on Nov 19, 2024

Update tokenizer_config.json

5c6ab95
verified

AlexHung29629 commited on Nov 19, 2024

Update config.json

7b1dee6
verified

AlexHung29629 commited on Nov 19, 2024

Upload Llama3ForCausalLM

a8aed0d
verified

AlexHung29629 commited on Nov 19, 2024

first commit

b48cee7

alex-ht commited on Nov 19, 2024

initial commit

4709cd5
verified

AlexHung29629 commited on Nov 19, 2024

Commit History

Upload Llama3ForCausalLM e7b1b15 verified

Update mllama_audio_model.py c13395b verified

Update modeling_llama3.py 5ab87b5 verified

Update tokenizer_config.json 5c6ab95 verified

Update config.json 7b1dee6 verified

Upload Llama3ForCausalLM a8aed0d verified

first commit b48cee7

initial commit 4709cd5 verified

Upload Llama3ForCausalLM

e7b1b15
verified

Update mllama_audio_model.py

c13395b
verified

Update modeling_llama3.py

5ab87b5
verified

Update tokenizer_config.json

5c6ab95
verified

Update config.json

7b1dee6
verified

Upload Llama3ForCausalLM

a8aed0d
verified

first commit

b48cee7

initial commit

4709cd5
verified