Instructions to use latimar/Phind-Codellama-34B-v2-exl2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use latimar/Phind-Codellama-34B-v2-exl2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="latimar/Phind-Codellama-34B-v2-exl2")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("latimar/Phind-Codellama-34B-v2-exl2")
model = AutoModelForCausalLM.from_pretrained("latimar/Phind-Codellama-34B-v2-exl2")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use latimar/Phind-Codellama-34B-v2-exl2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "latimar/Phind-Codellama-34B-v2-exl2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "latimar/Phind-Codellama-34B-v2-exl2",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/latimar/Phind-Codellama-34B-v2-exl2

SGLang

How to use latimar/Phind-Codellama-34B-v2-exl2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "latimar/Phind-Codellama-34B-v2-exl2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "latimar/Phind-Codellama-34B-v2-exl2",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "latimar/Phind-Codellama-34B-v2-exl2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "latimar/Phind-Codellama-34B-v2-exl2",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use latimar/Phind-Codellama-34B-v2-exl2 with Docker Model Runner:
```
docker model run hf.co/latimar/Phind-Codellama-34B-v2-exl2
```

What calibration data do u use?

by Apel-sin - opened Sep 14, 2023

Discussion

Apel-sin

Sep 14, 2023

Thanx for work! I think Phind-Codellama-34B-v2 it's the best local model ever :)
Please tell me what calibration data do you use?

latimar

Owner Sep 14, 2023

Yes, so far it's the best model for coding and general IT questions I've seen.
I'm using wikitext v2 raw, particularly this file: https://huggingface.co/datasets/wikitext/blob/refs%2Fconvert%2Fparquet/wikitext-2-raw-v1/test/0000.parquet

latimar changed discussion status to closed Sep 14, 2023

acrastt

Sep 15, 2023

Yes, so far it's the best model for coding and general IT questions I've seen.
I'm using wikitext v2 raw, particularly this file: https://huggingface.co/datasets/wikitext/blob/refs%2Fconvert%2Fparquet/wikitext-2-raw-v1/test/0000.parquet

Have you considered using a code dataset like The Stack(Deduped)?

latimar

Owner Sep 22, 2023

@acrastt I'd not expect huge differences from that. Only 100 rows of the dataset is used during conversion with the default settings of the exllamav2 convert script, anyways. And I'd imagine that the best calibration dataset for conversion is the validation part of the dataset the model was fine-tuned on. Unfortunately, phind's dataset is not open sourced.

acrastt

Sep 23, 2023

@acrastt I'd not expect huge differences from that. Only 100 rows of the dataset is used during conversion with the default settings of the exllamav2 convert script, anyways. And I'd imagine that the best calibration dataset for conversion is the validation part of the dataset the model was fine-tuned on. Unfortunately, phind's dataset is not open sourced.

Since it was fine-tuned for coding, I'd stick to a coding dataset to minimize the loss of coding abilities rather than general abilities.

latimar

Owner Oct 16, 2023

@acrastt Yep, it does make sense, but initially I did not think calibration dataset would matter much. But it seems that the 5-bpw-evol quant is better than the one based on wikitext (https://huggingface.co/latimar/Phind-Codellama-34B-v2-exl2/discussions/4) so I'm gonna make some more quants using different datasets...

latimar

Owner Oct 20, 2023

@acrastt Take a look at the new quants I made using megacode dataset, they seem to be much better: https://huggingface.co/latimar/Phind-Codellama-34B-v2-megacode-exl2

Apel-sin

May 1, 2024

Can you share a calibration dataset based on megacode?

latimar

Owner May 2, 2024

@Mandarin It's actually linked in the repo's README, and been there all the time -- https://huggingface.co/latimar/Phind-Codellama-34B-v2-megacode-exl2#datasets-used-for-calibration-and-ppl-measurement

https://huggingface.co/datasets/rombodawg/MegaCodeTraining

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment