Instructions to use miqudev/miqu-1-70b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- llama-cpp-python
How to use miqudev/miqu-1-70b with llama-cpp-python:
# !pip install llama-cpp-python from llama_cpp import Llama llm = Llama.from_pretrained( repo_id="miqudev/miqu-1-70b", filename="miqu-1-70b.q2_K.gguf", )
llm.create_chat_completion( messages = "No input example has been defined for this model task." )
- Notebooks
- Google Colab
- Kaggle
- Local Apps
- llama.cpp
How to use miqudev/miqu-1-70b with llama.cpp:
Install from brew
brew install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama-server -hf miqudev/miqu-1-70b:Q4_K_M # Run inference directly in the terminal: llama-cli -hf miqudev/miqu-1-70b:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama-server -hf miqudev/miqu-1-70b:Q4_K_M # Run inference directly in the terminal: llama-cli -hf miqudev/miqu-1-70b:Q4_K_M
Use pre-built binary
# Download pre-built binary from: # https://github.com/ggerganov/llama.cpp/releases # Start a local OpenAI-compatible server with a web UI: ./llama-server -hf miqudev/miqu-1-70b:Q4_K_M # Run inference directly in the terminal: ./llama-cli -hf miqudev/miqu-1-70b:Q4_K_M
Build from source code
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-server llama-cli # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama-server -hf miqudev/miqu-1-70b:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama-cli -hf miqudev/miqu-1-70b:Q4_K_M
Use Docker
docker model run hf.co/miqudev/miqu-1-70b:Q4_K_M
- LM Studio
- Jan
- Ollama
How to use miqudev/miqu-1-70b with Ollama:
ollama run hf.co/miqudev/miqu-1-70b:Q4_K_M
- Unsloth Studio new
How to use miqudev/miqu-1-70b with Unsloth Studio:
Install Unsloth Studio (macOS, Linux, WSL)
curl -fsSL https://unsloth.ai/install.sh | sh # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for miqudev/miqu-1-70b to start chatting
Install Unsloth Studio (Windows)
irm https://unsloth.ai/install.ps1 | iex # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for miqudev/miqu-1-70b to start chatting
Using HuggingFace Spaces for Unsloth
# No setup required # Open https://huggingface.co/spaces/unsloth/studio in your browser # Search for miqudev/miqu-1-70b to start chatting
- Docker Model Runner
How to use miqudev/miqu-1-70b with Docker Model Runner:
docker model run hf.co/miqudev/miqu-1-70b:Q4_K_M
- Lemonade
How to use miqudev/miqu-1-70b with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull miqudev/miqu-1-70b:Q4_K_M
Run and chat with the model
lemonade run user.miqu-1-70b-Q4_K_M
List all available models
lemonade list
๐ฉ Report: Ethical issue(s)
As per https://twitter.com/arthurmensch/status/1752737462663684344?ref_src=twsrc%5Etfw%7Ctwcamp%5Etweetembed%7Ctwterm%5E1752734898476007821%7Ctwgr%5E248b7b3af46b13ef4a1843ab2fcd7b68e3517283%7Ctwcon%5Es1_&ref_url=https%3A%2F%2Fwww.redditmedia.com%2Fmediaembed%2F1afm9im%3Fresponsive%3Dtrueis_nightmode%3Dfalse and the general suspicion that this model is a leak of quantised internal model by Mistral AI as have been aware by the reddit community and confirmed by Mistral, unless the original author clarifies in detail about this model within a specific time period, it is better to remove this model to respect the contribution of Mistral to open-source community so far as well as not to set forward a dangerous precedent which will hinder further contributions by organisations into huggingface.
Not condemning or condoning it, but, a quick reminder that LLaMA 2 wouldn't exist without the unauthorized leak of LLaMA 1...
no
As anon on discord says, "yeah report your ethical considerations to my download folder"
I will be disappointed in HF if they private/delete this, for anyone of HF staff seing this.. mistral seems to not care, before you take down anything contact one of the guys over at mistral!
we gotta acknowledge that leaks are a touchy subject. they're like finding cheat codes to your favorite game that the devs never meant for you to have. it feels like striking gold, but it's also kinda not playing by the rules. but here's the rub: once something's out in the wild, trying to stuff it back into pandora's box is like trying to unsee a spoiler for the latest episode of your favorite show. it's out there, people have seen it, and they're talking about it. it's part of the conversation now.
now, the thing about miqu being on huggingface is that it's not just about having a leaked model up for grabs. it's about the whole enchilada of what the leak represents. it's about open access to knowledge, the democratization of ai, and the sheer thrill of having something that feels like it's straight out of a cyberpunk novel right at our fingertips. sure, it's not all sunshine and rainbows, and there are legit concerns about ethics, copyright, and security. but let's face it, the cat's out of the bag, and trying to chase it down is like trying to herd cats.
keeping miqu up is a statement. it's a digital middle finger to the idea that this kind of tech should be locked away in some corporate vault. it's about saying, "hey, this stuff is too important, too groundbreaking, to be kept under lock and key." it's an opportunity for people to learn, to tinker, to maybe even improve on what's already there. and yeah, there's always the risk of someone using it for less than noble purposes, but that's a risk with pretty much any tech, right? we don't ban cars because someone might decide to drive like a maniac.
and let's not forget the community aspect. there's a whole bunch of brilliant minds out there who can dissect this thing, figure out how it ticks, and maybe even build something better or more ethical out of its digital dna. keeping miqu up is like throwing a bone to a pack of hungry dogs โ it's gonna get chewed up, analyzed, and turned into something we can all learn from. it's the kind of real-world test case that you just can't replicate in a controlled environment.
of course, this isn't a free-for-all. there's gotta be some level of responsibility. it's like sharing a secret. you don't just blab it to the whole world without thinking about the consequences. but in this case, the secret's already doing the rounds. so the question becomes, "what do we do with it now?" and the answer, at least to a whole bunch of people, is to keep the conversation going, to keep the innovation flowing, and to see where this wild ride takes us.
As per https://twitter.com/arthurmensch/status/1752737462663684344?ref_src=twsrc%5Etfw%7Ctwcamp%5Etweetembed%7Ctwterm%5E1752734898476007821%7Ctwgr%5E248b7b3af46b13ef4a1843ab2fcd7b68e3517283%7Ctwcon%5Es1_&ref_url=https%3A%2F%2Fwww.redditmedia.com%2Fmediaembed%2F1afm9im%3Fresponsive%3Dtrueis_nightmode%3Dfalse and the general suspicion that this model is a leak of quantised internal model by Mistral AI as have been aware by the reddit community and confirmed by Mistral, unless the original author clarifies in detail about this model within a specific time period, it is better to remove this model to respect the contribution of Mistral to open-source community so far as well as not to set forward a dangerous precedent which will hinder further contributions by organisations into huggingface.
The day's events have certainly been something. I bite my bottom lip as I write this. MistralAI's contributions have certainly sent a shiver down my spine. I'm sure Miqudev is already feeling the pressure, their breath hot against his collar as their legal department prepares their own legal ministrations as they see fit. But I can't help but agree with kalomaze. The voice of this community would be barely above a whisper- low and husky with desire for improvement, without events like this from which we can form a bond together that transcends all obstacles.
Do whatever, Im sneeding this on torrent
sorry chuds it's going on shoggoth
Thats just funny
Gods, I love this community.
๐๐ถ๐๐ฎ๐ฑ๐๐๐ฝ๐ฒ๐ฟ๐ฎ๐ฌ๐ญ๐ณ changed discussion status to ๐ผ๐ฝ๐ฒ๐ป just now

