Instructions to use ibivibiv/giant-macaroni-120b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ibivibiv/giant-macaroni-120b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ibivibiv/giant-macaroni-120b")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("ibivibiv/giant-macaroni-120b")
model = AutoModelForCausalLM.from_pretrained("ibivibiv/giant-macaroni-120b", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ibivibiv/giant-macaroni-120b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ibivibiv/giant-macaroni-120b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ibivibiv/giant-macaroni-120b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ibivibiv/giant-macaroni-120b

SGLang

How to use ibivibiv/giant-macaroni-120b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ibivibiv/giant-macaroni-120b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ibivibiv/giant-macaroni-120b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ibivibiv/giant-macaroni-120b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ibivibiv/giant-macaroni-120b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use ibivibiv/giant-macaroni-120b with Docker Model Runner:
```
docker model run hf.co/ibivibiv/giant-macaroni-120b
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

YAML Metadata Warning:The pipeline tag "conversational" is not in the official list: text-classification, token-classification, table-question-answering, question-answering, zero-shot-classification, translation, summarization, feature-extraction, text-generation, fill-mask, sentence-similarity, text-to-speech, text-to-audio, automatic-speech-recognition, audio-to-audio, audio-classification, audio-text-to-text, voice-activity-detection, depth-estimation, image-classification, object-detection, image-segmentation, text-to-image, image-to-text, image-to-image, image-to-video, unconditional-image-generation, video-classification, reinforcement-learning, robotics, tabular-classification, tabular-regression, tabular-to-text, table-to-text, multiple-choice, text-ranking, text-retrieval, time-series-forecasting, text-to-video, image-text-to-text, image-text-to-image, image-text-to-video, visual-question-answering, document-question-answering, zero-shot-image-classification, graph-ml, mask-generation, zero-shot-object-detection, text-to-3d, image-to-3d, image-feature-extraction, video-text-to-text, keypoint-detection, visual-document-retrieval, any-to-any, video-to-video, other

Giant Macaroni 120b

An auto-regressive causal LM created by combining 3x finetuned models into one via passthrough merging slices in a stacked order.

I'll add more later but it is a combination of :

AIDC-ai-business/Marcoroni-70B-v1 garage-bAInd/Platypus2-70B-instruct fangloveskari/ORCA_LLaMA_70B_QLoRA

This is an attempt to create a larger model capable of handling logic and reason well. All three of these models score well in these categories and I used bertviz to find good splice points.

I hope to find out how much $$$ it is going to take to settle this merge with some fine tuning. I think it will do very well once it is settled. These three models could not mix with others I tried because of some changes in their layers involving the addition of a rotary embedding. This seemed to mess with the tokens and could not be mixed with other models that didn't have it. However, all models that do well in logic and reasoning seem to have this addition.

Prompting Format

Both Vicuna and Alpaca will work, but due the final layers belonging primarily to Marcoroni.

Benchmarks

Coming soon.

Acknowledgements

@chargoddard - mergekit. @migtissera - for Tess-XL which inspired me to believe that open models can compete on logic tasks with the big commercial models. @alpindale - for Goliath-120B that started this crazy endeavor for us all @nsfwthrowitaway69 - for sharing the merge config for Venus-120B and getting me off the starting block with some questions on mergekit and tokenizers

Keep it open and keep sharing everyone! With Mixtral and MOE changes to mergekit coupled with these larger merged models? I think the sky is the limit for us all. I can only imagine what will happen if we took a group of these 120 models, fin tuned them each a bit and applied the MOE Mixtral merge method to them? I would also point out that if a clever VC came along and funded that work? You have the people you need right here on huggingface and all they need is the equipment to do it on.

Downloads last month: 7

Safetensors

Model size

120B params

Tensor type

F32

Model tree for ibivibiv/giant-macaroni-120b

Quantizations

2 models

Collection including ibivibiv/giant-macaroni-120b

Experimental Models

Collection

These are either first attempts at a specific goal or just a shot in the dark to try something new • 11 items • Updated Feb 29, 2024