Instructions to use SandLogicTechnologies/Phi-3-medium-128k-instruct-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use SandLogicTechnologies/Phi-3-medium-128k-instruct-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf SandLogicTechnologies/Phi-3-medium-128k-instruct-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf SandLogicTechnologies/Phi-3-medium-128k-instruct-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf SandLogicTechnologies/Phi-3-medium-128k-instruct-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf SandLogicTechnologies/Phi-3-medium-128k-instruct-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf SandLogicTechnologies/Phi-3-medium-128k-instruct-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf SandLogicTechnologies/Phi-3-medium-128k-instruct-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf SandLogicTechnologies/Phi-3-medium-128k-instruct-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf SandLogicTechnologies/Phi-3-medium-128k-instruct-GGUF:Q4_K_M

Use Docker

docker model run hf.co/SandLogicTechnologies/Phi-3-medium-128k-instruct-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use SandLogicTechnologies/Phi-3-medium-128k-instruct-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "SandLogicTechnologies/Phi-3-medium-128k-instruct-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "SandLogicTechnologies/Phi-3-medium-128k-instruct-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/SandLogicTechnologies/Phi-3-medium-128k-instruct-GGUF:Q4_K_M

Ollama
How to use SandLogicTechnologies/Phi-3-medium-128k-instruct-GGUF with Ollama:
```
ollama run hf.co/SandLogicTechnologies/Phi-3-medium-128k-instruct-GGUF:Q4_K_M
```

Unsloth Studio

How to use SandLogicTechnologies/Phi-3-medium-128k-instruct-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for SandLogicTechnologies/Phi-3-medium-128k-instruct-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for SandLogicTechnologies/Phi-3-medium-128k-instruct-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for SandLogicTechnologies/Phi-3-medium-128k-instruct-GGUF to start chatting

Docker Model Runner
How to use SandLogicTechnologies/Phi-3-medium-128k-instruct-GGUF with Docker Model Runner:
```
docker model run hf.co/SandLogicTechnologies/Phi-3-medium-128k-instruct-GGUF:Q4_K_M
```

Lemonade

How to use SandLogicTechnologies/Phi-3-medium-128k-instruct-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull SandLogicTechnologies/Phi-3-medium-128k-instruct-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Phi-3-medium-128k-instruct-GGUF-Q4_K_M

List all available models

lemonade list

Atomic Chat

Phi-3-Medium-128K-Instruct

Phi-3-Medium-128K-Instruct is a 14-billion-parameter instruction-tuned language model developed by Microsoft. It is designed to deliver strong reasoning, structured responses, and long-context comprehension across a wide range of tasks. With support for up to 128K tokens of context, this model is well suited for document analysis, extended conversations, and complex multi-step workflows.

This repository provides access to the instruction-optimized version of Phi-3 Medium with extended context capabilities for research and deployment.

Model Overview

Model Name: Phi-3-Medium-128K-Instruct
Base Model: microsoft/Phi-3-medium-128k-instruct
Architecture: Decoder-only Transformer
Parameter Count: 14 Billion
Context Length: 128K tokens
Modalities: Text
Developer: Microsoft
License: MIT License

Model Variants

Instruction-Tuned Version

Optimized for conversational and task-oriented prompts
Improved adherence to structured instructions
Enhanced reasoning and step-by-step explanation capabilities
Designed for safe and helpful outputs

Quantization Details

Q4_K_M

Approx. ~71% size reduction
Very low memory footprint (~7.98 GB)
Optimized for CPU inference and low-VRAM GPUs
Faster token generation speeds
Minor degradation in complex analytical or long-chain reasoning tasks

Q5_K_M

Approx. ~66% size reduction
Better fidelity to the original FP16 model (9.38 GB)
Improved coherence and reasoning consistency
Recommended when slightly more memory is available

Training Background

Phi-3-Medium-128K-Instruct is built on the Phi-3 architecture and trained to balance efficiency and performance. Pretrained on carefully curated data with emphasis on reasoning, coding, and structured problem-solving, followed by instruction tuning to improve prompt adherence, response clarity, and real-world usability.

Pretraining

Trained on a curated mixture of high-quality text sources
Focused on reasoning, coding, mathematics, and general knowledge
Optimized using autoregressive next-token prediction

Instruction Tuning

Fine-tuned on instruction-following datasets
Enhanced performance on question answering, summarization, coding, and dialogue
Improved alignment for clarity, relevance, and structured outputs

Key Capabilities

Long-context understanding
Supports up to 128K tokens, enabling large document processing and persistent conversational memory.
Advanced reasoning
Performs multi-step logical reasoning, analytical problem solving, and structured explanations.
Code generation and analysis
Capable of generating, debugging, and explaining code across multiple programming languages.
Conversational AI
Maintains coherence across extended multi-turn interactions.
Document summarization and extraction
Handles long reports, contracts, and research papers effectively.

Usage Example

llama.cpp

./llama-cli \
  -m SandlogicTechnologies\Phi-3-mini-4k-instruct_Q4_K_M.gguf \
  -p "Explain transformers in simple terms."

Recommended Applications

Enterprise document processing :- Analyze long-form documents, compliance materials, and technical manuals.
Research and experimentation :- Evaluate reasoning performance and long-context capabilities.
Code assistance tools :- Integrate into development environments for coding support.
Conversational AI systems :- Deploy in chatbots requiring extended context memory.
Educational tools :- Generate structured explanations and step-by-step solutions.

Deployment Considerations

Requires sufficient GPU memory for optimal performance
Mixed precision (FP16/BF16) recommended for efficiency
Suitable for distributed or high-memory inference setups
Ensure adherence to the MIT license terms when deploying commercially

Acknowledgments

These quantized models are based on the original work by Microsoft development team.

Special thanks to:

The Microsoft team for developing and releasing the Phi-3-mini-4k-instruct model.
Georgi Gerganov and the entire llama.cpp open-source community for enabling efficient model quantization and inference via the GGUF format.

Contact

For any inquiries or support, please contact us at support@sandlogic.com or visit our Website.

Downloads last month: 10

GGUF

Model size

14B params

Architecture

phi3

Hardware compatibility

4-bit

5-bit

Model tree for SandLogicTechnologies/Phi-3-medium-128k-instruct-GGUF

Base model

microsoft/Phi-3-medium-128k-instruct

Quantized

(75)

this model