Instructions to use WizardLMTeam/WizardLM-70B-V1.0 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use WizardLMTeam/WizardLM-70B-V1.0 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="WizardLMTeam/WizardLM-70B-V1.0")# Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("WizardLMTeam/WizardLM-70B-V1.0") model = AutoModelForCausalLM.from_pretrained("WizardLMTeam/WizardLM-70B-V1.0") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use WizardLMTeam/WizardLM-70B-V1.0 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "WizardLMTeam/WizardLM-70B-V1.0" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "WizardLMTeam/WizardLM-70B-V1.0", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/WizardLMTeam/WizardLM-70B-V1.0
- SGLang
How to use WizardLMTeam/WizardLM-70B-V1.0 with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "WizardLMTeam/WizardLM-70B-V1.0" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "WizardLMTeam/WizardLM-70B-V1.0", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "WizardLMTeam/WizardLM-70B-V1.0" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "WizardLMTeam/WizardLM-70B-V1.0", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use WizardLMTeam/WizardLM-70B-V1.0 with Docker Model Runner:
docker model run hf.co/WizardLMTeam/WizardLM-70B-V1.0
dataset
Please release the dataset used to train this model.
Would love for this to be a truly open-source model, as it used to be.
It's very sad and unfortunate that @WizardLM hasn't released the data for 1.1 or 1.2 or WizardCoder.
Similar to when OpenAI decided to become closed.
Please change your mind and start releasing your datasets as well as your models.
The community will greatly appreciate this. We love our open-source community and very sad to lose WizardLM to closed-source.
Hi,
Recently, there have been clear changes in the open-source policy and regulations of our overall organization's code, data, and models.
Despite this, we have still worked hard to obtain opening the weights of the model first, but the data involves stricter auditing and is in review with our legal team .
Our researchers have no authority to publicly release them without authorization.
Thank you for your understanding.
@WizardLM Here's an email written by Llama 2 70B:
Hello WizardLM,
I understand that you are unable to release the dataset used to train your model due to legal restrictions. However, I would like to suggest a possible solution that could benefit both your organization and the open-source community.
Have you considered releasing a subset of the dataset, or a modified version of the dataset that removes any sensitive information? This would allow the community to still benefit from the work that you have done, while also respecting any legal or ethical restrictions that you may have.
Additionally, you could consider providing more information about the data that you are using, such as the source of the data, the format of the data, and any preprocessing steps that you have applied. This would allow the community to better understand how the model was trained, and potentially even contribute to the development of the model.
I hope that this suggestion is helpful, and I look forward to hearing your thoughts on the matter.
Best regards, acrastt.