Instructions to use androlike/astramix_l2_7b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use androlike/astramix_l2_7b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="androlike/astramix_l2_7b")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("androlike/astramix_l2_7b")
model = AutoModelForCausalLM.from_pretrained("androlike/astramix_l2_7b", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use androlike/astramix_l2_7b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "androlike/astramix_l2_7b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "androlike/astramix_l2_7b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/androlike/astramix_l2_7b

SGLang

How to use androlike/astramix_l2_7b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "androlike/astramix_l2_7b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "androlike/astramix_l2_7b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "androlike/astramix_l2_7b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "androlike/astramix_l2_7b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use androlike/astramix_l2_7b with Docker Model Runner:
```
docker model run hf.co/androlike/astramix_l2_7b
```

Introduction

Astramix is a merge of various Llama-2-7b finetune models, using ties-merge method, thanks for Chargoddard. Subsequently, lora merge script was used, created by zarakiquemparte (link to the Hugging Face profile).

Approximate rating for this model, noticed in short-time use:

Great roleplay capabilities, limited only by model parameters.
Poor quality reasoning, because of the model limitations.
Almost censorship absence (yet some parts can be generated in the output).

Feel free to test the model.

Following base model used for merge: Llama-2-7B-fp16

List of models used for merge:

Then, two LoRAs was merged into basemix model, using script specified above:

limarp-llama2-v2 (Licensed under AGPLv3)
airoboros-lmoe-7b-2.1 (Utilizing creative version)

Here is quantized versions of the model:

I suggest using Alpaca instruct format:

### Instruction:
(your instruct prompt is here)
### Response:

Limitations and risks

Llama2 and its derivatives (finetunes) is licensed under LLama 2 Community License, various finetunes or (Q)LoRAs has appropriate licenses depending on used datasets in finetuning or training Low-Rank Adaptations. This mix can generate heavily biased output, which aren't suitable for minors or common audience due to using limarp in the merge.

Downloads last month: 11