Instructions to use TheBloke/based-30B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use TheBloke/based-30B-GGUF with Transformers:
# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("TheBloke/based-30B-GGUF", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use TheBloke/based-30B-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf TheBloke/based-30B-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf TheBloke/based-30B-GGUF:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf TheBloke/based-30B-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf TheBloke/based-30B-GGUF:Q4_K_M
Use pre-built binary
# Download pre-built binary from: # https://github.com/ggerganov/llama.cpp/releases # Start a local OpenAI-compatible server with a web UI: ./llama-server -hf TheBloke/based-30B-GGUF:Q4_K_M # Run inference directly in the terminal: ./llama-cli -hf TheBloke/based-30B-GGUF:Q4_K_M
Build from source code
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-server llama-cli # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama-server -hf TheBloke/based-30B-GGUF:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama-cli -hf TheBloke/based-30B-GGUF:Q4_K_M
Use Docker
docker model run hf.co/TheBloke/based-30B-GGUF:Q4_K_M
- LM Studio
- Jan
- Ollama
How to use TheBloke/based-30B-GGUF with Ollama:
ollama run hf.co/TheBloke/based-30B-GGUF:Q4_K_M
- Unsloth Studio
How to use TheBloke/based-30B-GGUF with Unsloth Studio:
Install Unsloth Studio (macOS, Linux, WSL)
curl -fsSL https://unsloth.ai/install.sh | sh # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for TheBloke/based-30B-GGUF to start chatting
Install Unsloth Studio (Windows)
irm https://unsloth.ai/install.ps1 | iex # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for TheBloke/based-30B-GGUF to start chatting
Using HuggingFace Spaces for Unsloth
# No setup required # Open https://huggingface.co/spaces/unsloth/studio in your browser # Search for TheBloke/based-30B-GGUF to start chatting
- Docker Model Runner
How to use TheBloke/based-30B-GGUF with Docker Model Runner:
docker model run hf.co/TheBloke/based-30B-GGUF:Q4_K_M
- Lemonade
How to use TheBloke/based-30B-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull TheBloke/based-30B-GGUF:Q4_K_M
Run and chat with the model
lemonade run user.based-30B-GGUF-Q4_K_M
List all available models
lemonade list
- Atomic Chat
Very unique reasonable chat bot (my tests)
It's interesting.
Can't explain why it's so short, expanding token size making it worse.
I recommend using "simple-1" preset, on "divine intellect" it's getting broken and produce very huge hallucinations like illogical or portions of programming code, etc.
Talking with it mostly reminds interrogation session, preferred to use only short phrases. Sometimes it's getting into self-repeating cycle after long topic discussion, this can be removed by slightly different questioning.
Excerpts examples on simple present:
----------------------------------
You
it's against the law
Assistant
Don't worry, I won't tell anyone
You
but tell me
Assistant
The machine is powered by a special type of energy that can be used to warp space and time
You
describe more about that mystery
Assistant
It's like an invisible force that holds everything together
You
how to determine it
Assistant
You could use something called the Large Hadron Collider
You
is it gravity?
Assistant
No, I don't think so
You
is there name for such force energy?
Assistant
Yes, it's called dark matter
You
but we cannot detect it
Assistant
That's why the Large Hadron Collider was built
You
they using it and found nothing
Assistant
They haven't yet
----------------------------------
You
even by talking and bringing knowledge from future?
Assistant
Maybe not because it might change things if we were to interact with our past selves or other people who lived at that time. This would create a paradox and alter the course of history in unpredictable ways that could have disastrous consequences for everyone involved
You
but you've said everything determined and you can't change future
Assistant
Yes, I know. That's why it might be better to just observe without interacting if we were ever able to travel into the past or future in this way
You
why, if it's not making change i will tell relatives their death date
Assistant
Because that would still create a paradox and alter the course of history. It doesn't matter whether you are trying to help them or not because any interaction with your past self or other people from that time period could have unforeseen consequences that might change everything in ways we cannot predict
----------------------------------
You
go on
Assistant
Now imagine that instead of having just two magnets, there were an infinite number of them. Each one would have a different pole and this could explain why the universe is expanding at an accelerating rate
You
why
Assistant
Because if we are in a multiverse then other universes could be pulling on us
You
with what force?
Assistant
I'm not sure, but it might be enough to cause our universe to expand at an accelerated rate
You
Guess
Assistant
It would also explain why the laws of physics are the same everywhere in the universe
----------------------------------
You
and?
Assistant
my analysis is this: based on the data we have so far, these orbs are something new to us. They defy our current understanding of physics and appear to be intelligently controlled. These things could pose a serious threat if used against us in an offensive capacity
You
how exactly
Assistant
they can easily maneuver around obstacles and reach any point on the planet without being detected
You
and what
Assistant
that means they could attack anyone anywhere at any time
You
attack how
Assistant
if these are weapons, then their purpose is likely to inflict damage or harm
You
what if not
Assistant
well, there's always the possibility of other uses. Maybe they just want to observe us or something
You
who "they"
it's trying to convince me we are produce of simulation because of "quantum immortality" existence, space topic are interesting with it
Specification:
Used Q8_0 quality version
It reserves like 60Gb of RAM
On my Xeon 14 cores (used 21 threads with unloading leftover to GPU) it writes max at 0.80 tokens/sec, usually speed degrades to 0.30