Instructions to use anton-l/gpt-j-tiny-random with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use anton-l/gpt-j-tiny-random with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="anton-l/gpt-j-tiny-random")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("anton-l/gpt-j-tiny-random")
model = AutoModelForCausalLM.from_pretrained("anton-l/gpt-j-tiny-random")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use anton-l/gpt-j-tiny-random with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "anton-l/gpt-j-tiny-random"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "anton-l/gpt-j-tiny-random",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/anton-l/gpt-j-tiny-random

SGLang

How to use anton-l/gpt-j-tiny-random with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "anton-l/gpt-j-tiny-random" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "anton-l/gpt-j-tiny-random",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "anton-l/gpt-j-tiny-random" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "anton-l/gpt-j-tiny-random",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use anton-l/gpt-j-tiny-random with Docker Model Runner:
```
docker model run hf.co/anton-l/gpt-j-tiny-random
```

rust_weights

by lerouxrgd - opened Oct 16, 2022

base: refs/heads/main

←

from: refs/pr/1

Discussion Files changed

-0

lerouxrgd

Oct 16, 2022

•

edited Oct 16, 2022

Add rust_model.ot model weights.

This is particularly of use for testing GPT-J implementation in Rust.
See related PR

Add rust weightsabb07a55

lerouxrgd

Oct 16, 2022

•

edited Oct 16, 2022

Beside, is this model licensed under the Apache 2.0 like gpt-j-6b from EleutherAI ?

lerouxrgd changed pull request status to open Oct 16, 2022

lerouxrgd

Oct 21, 2022

@anton-l Just pinging you as I am not sure whether you get notifications when a PR is opened (If you do, sorry for the spam !)

anton-l

Owner Oct 21, 2022

@lerouxrgd maybe it makes sense to just set up another repo for the rust weights? Because the model here is literally just a dummy one, with a minimally working config and randomly initialized weights 😅 It was intended for unit-testing the transformers implementation

lerouxrgd

Oct 21, 2022

Yes I am well aware that the weights are literally random :D But this is fine, as it would serve the same purpose that is unit testing but for the Rust implementation. It's actually quite useful to have small weights for such tests, and I think that having the Python/Rust weights together makes it clear that the unit tests are correct (I intend to compare the final logits in Rust with the final logits in Python, when using the same model weights).

For some other models the Rust weights are being stored in the same repo:
https://huggingface.co/sentence-transformers/distiluse-base-multilingual-cased/discussions/1
I intend to add the Rust weights to EleutherAI's gpt-j-6b repo too once the unit tests are complete.

anton-l changed pull request status to merged Oct 24, 2022

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment