Instructions to use WhiteRabbitNeo/Trinity-13B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use WhiteRabbitNeo/Trinity-13B with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="WhiteRabbitNeo/Trinity-13B", trust_remote_code=True)# Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("WhiteRabbitNeo/Trinity-13B", trust_remote_code=True) model = AutoModelForCausalLM.from_pretrained("WhiteRabbitNeo/Trinity-13B", trust_remote_code=True) - Inference
- Notebooks
- Google Colab
- Kaggle
- Local Apps
- vLLM
How to use WhiteRabbitNeo/Trinity-13B with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "WhiteRabbitNeo/Trinity-13B" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "WhiteRabbitNeo/Trinity-13B", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/WhiteRabbitNeo/Trinity-13B
- SGLang
How to use WhiteRabbitNeo/Trinity-13B with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "WhiteRabbitNeo/Trinity-13B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "WhiteRabbitNeo/Trinity-13B", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "WhiteRabbitNeo/Trinity-13B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "WhiteRabbitNeo/Trinity-13B", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use WhiteRabbitNeo/Trinity-13B with Docker Model Runner:
docker model run hf.co/WhiteRabbitNeo/Trinity-13B
Commit ·
b03b56d
1
Parent(s): 990ccc5
Update README.md
Browse files
README.md
CHANGED
|
@@ -98,3 +98,72 @@ while True:
|
|
| 98 |
|
| 99 |
```
|
| 100 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 98 |
|
| 99 |
```
|
| 100 |
|
| 101 |
+
|
| 102 |
+
# Sample Conversation:
|
| 103 |
+
|
| 104 |
+
|
| 105 |
+
You: Code me a Fast API server, with one endpoint. This endpoint returns files from a S3 bucket.
|
| 106 |
+
|
| 107 |
+
Here is how to code a Fast API server with one endpoint that returns files from an S3 bucket:
|
| 108 |
+
|
| 109 |
+
```python
|
| 110 |
+
from fastapi import FastAPI
|
| 111 |
+
from starlette.responses import StreamingResponse
|
| 112 |
+
import boto3
|
| 113 |
+
import botocore.exceptions
|
| 114 |
+
|
| 115 |
+
app = FastAPI()
|
| 116 |
+
s3_client = boto3.client('s3')
|
| 117 |
+
|
| 118 |
+
@app.get("/s3files")
|
| 119 |
+
async def get_s3_files(bucket_name: str, key: str):
|
| 120 |
+
try:
|
| 121 |
+
obj = s3_client.get_object(Bucket=bucket_name, Key=key)
|
| 122 |
+
return StreamingResponse(obj['Body'], media_type=obj['ContentType'])
|
| 123 |
+
except botocore.exceptions.ClientError as e:
|
| 124 |
+
return {"error": str(e)}, 404
|
| 125 |
+
```
|
| 126 |
+
|
| 127 |
+
To explain:
|
| 128 |
+
- We initialize a FastAPI app and an S3 client
|
| 129 |
+
- The `/s3files` endpoint takes the bucket name and object key as parameters
|
| 130 |
+
- We try to get the object from S3 using the S3 client
|
| 131 |
+
- If successful, we return the object's Body wrapped in a StreamingResponse, setting the ContentType
|
| 132 |
+
- If the client raises an error, we return a 404 with the error message
|
| 133 |
+
|
| 134 |
+
This allows clients to request files from an S3 bucket by specifying the bucket name and object key. The endpoint streams the file content back to the client.
|
| 135 |
+
|
| 136 |
+
To elaborate further:
|
| 137 |
+
- The endpoint can be extended to validate inputs, handle authentication, logging, etc.
|
| 138 |
+
- We could make this endpoint more generic by accepting a list of object keys instead of requiring a single key.
|
| 139 |
+
- There are several ways to optimize performance, like caching responses.
|
| 140 |
+
|
| 141 |
+
Let me know if you have any other questions!
|
| 142 |
+
|
| 143 |
+
You: How do I authenticate s3?
|
| 144 |
+
|
| 145 |
+
Here are a few ways to authenticate with S3:
|
| 146 |
+
|
| 147 |
+
1. Credentials - You can provide AWS access key ID and secret access key to boto3. This will sign requests automatically.
|
| 148 |
+
|
| 149 |
+
```python
|
| 150 |
+
import boto3
|
| 151 |
+
|
| 152 |
+
s3 = boto3.client('s3',
|
| 153 |
+
aws_access_key_id='YOUR_ACCESS_KEY',
|
| 154 |
+
aws_secret_access_key='YOUR_SECRET_KEY')
|
| 155 |
+
```
|
| 156 |
+
|
| 157 |
+
2. IAM roles - If you are running on EC2 with an IAM role, you don't need to provide credentials. Boto3 will automatically use the role.
|
| 158 |
+
|
| 159 |
+
3. Environment variables - Store credentials in environment variables and read them in boto3.
|
| 160 |
+
|
| 161 |
+
4. AWS credentials file - Store credentials in ~/.aws/credentials. Boto3 will automatically use them.
|
| 162 |
+
|
| 163 |
+
5. STS Assume roles - Get temporary credentials from STS and assume a role to access S3.
|
| 164 |
+
|
| 165 |
+
6. AWS SDK credential providers - Use environment variables, configuration files, etc to load credentials.
|
| 166 |
+
|
| 167 |
+
So in summary, the best way depends on your deployment environment. The best practice is to use IAM roles when possible. Other options like environment variables and credential files can be useful for testing.
|
| 168 |
+
|
| 169 |
+
Let me know if you have any other questions!
|