Text Generation
Transformers
Safetensors
GGUF
English
mistral
medical
spinal-cord-injury
healthcare
disability
accessibility
fine-tuned
lora
conversational
text-generation-inference
Instructions to use basiphobe/sci-assistant with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use basiphobe/sci-assistant with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="basiphobe/sci-assistant") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("basiphobe/sci-assistant") model = AutoModelForCausalLM.from_pretrained("basiphobe/sci-assistant", device_map="auto") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use basiphobe/sci-assistant with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf basiphobe/sci-assistant:Q4_K_M # Run inference directly in the terminal: llama cli -hf basiphobe/sci-assistant:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf basiphobe/sci-assistant:Q4_K_M # Run inference directly in the terminal: llama cli -hf basiphobe/sci-assistant:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf basiphobe/sci-assistant:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf basiphobe/sci-assistant:Q4_K_M
Use Docker
docker model run hf.co/basiphobe/sci-assistant:Q4_K_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use basiphobe/sci-assistant with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "basiphobe/sci-assistant" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "basiphobe/sci-assistant", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/basiphobe/sci-assistant:Q4_K_M
- SGLang
How to use basiphobe/sci-assistant with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "basiphobe/sci-assistant" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "basiphobe/sci-assistant", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "basiphobe/sci-assistant" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "basiphobe/sci-assistant", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Ollama
How to use basiphobe/sci-assistant with Ollama:
ollama run hf.co/basiphobe/sci-assistant:Q4_K_M
- Unsloth Desktop
- Docker Model Runner
How to use basiphobe/sci-assistant with Docker Model Runner:
docker model run hf.co/basiphobe/sci-assistant:Q4_K_M
- Lemonade
How to use basiphobe/sci-assistant with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull basiphobe/sci-assistant:Q4_K_M
Run and chat with the model
lemonade run user.sci-assistant-Q4_K_M
List all available models
lemonade list
- Atomic Chat
Update README with both LoRA and merged model usage options
Browse files
README.md
CHANGED
|
@@ -22,11 +22,33 @@ The model was trained on curated SCI-related content including:
|
|
| 22 |
|
| 23 |
## Usage
|
| 24 |
|
|
|
|
|
|
|
|
|
|
| 25 |
```python
|
| 26 |
from transformers import AutoModelForCausalLM, AutoTokenizer
|
| 27 |
|
| 28 |
-
model = AutoModelForCausalLM.from_pretrained("
|
| 29 |
-
tokenizer = AutoTokenizer.from_pretrained("
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 30 |
|
| 31 |
# Example usage
|
| 32 |
prompt = "What are the signs of autonomic dysreflexia?"
|
|
@@ -48,6 +70,12 @@ response = tokenizer.decode(outputs[0], skip_special_tokens=True)
|
|
| 48 |
- Not a replacement for professional medical care
|
| 49 |
- May not reflect the most recent medical developments
|
| 50 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 51 |
## Technical Details
|
| 52 |
|
| 53 |
- **Base Model**: teknium/OpenHermes-2.5-Mistral-7B
|
|
|
|
| 22 |
|
| 23 |
## Usage
|
| 24 |
|
| 25 |
+
This repository contains both the LoRA adapter and the full merged model. Choose the option that works best for you:
|
| 26 |
+
|
| 27 |
+
### Option 1: Use the Full Merged Model (Recommended)
|
| 28 |
```python
|
| 29 |
from transformers import AutoModelForCausalLM, AutoTokenizer
|
| 30 |
|
| 31 |
+
model = AutoModelForCausalLM.from_pretrained("basiphobe/sci-assistant")
|
| 32 |
+
tokenizer = AutoTokenizer.from_pretrained("basiphobe/sci-assistant")
|
| 33 |
+
|
| 34 |
+
# Example usage
|
| 35 |
+
prompt = "What are the signs of autonomic dysreflexia?"
|
| 36 |
+
inputs = tokenizer(prompt, return_tensors="pt")
|
| 37 |
+
outputs = model.generate(**inputs, max_length=200)
|
| 38 |
+
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
|
| 39 |
+
```
|
| 40 |
+
|
| 41 |
+
### Option 2: Use the LoRA Adapter (Smaller Download)
|
| 42 |
+
```python
|
| 43 |
+
from transformers import AutoModelForCausalLM, AutoTokenizer
|
| 44 |
+
from peft import PeftModel
|
| 45 |
+
|
| 46 |
+
# Load base model
|
| 47 |
+
base_model = AutoModelForCausalLM.from_pretrained("teknium/OpenHermes-2.5-Mistral-7B")
|
| 48 |
+
tokenizer = AutoTokenizer.from_pretrained("teknium/OpenHermes-2.5-Mistral-7B")
|
| 49 |
+
|
| 50 |
+
# Load LoRA adapter
|
| 51 |
+
model = PeftModel.from_pretrained(base_model, "basiphobe/sci-assistant")
|
| 52 |
|
| 53 |
# Example usage
|
| 54 |
prompt = "What are the signs of autonomic dysreflexia?"
|
|
|
|
| 70 |
- Not a replacement for professional medical care
|
| 71 |
- May not reflect the most recent medical developments
|
| 72 |
|
| 73 |
+
## Files in this Repository
|
| 74 |
+
|
| 75 |
+
- **Full Merged Model**: Ready-to-use model files (`model-*.safetensors`, `config.json`, etc.)
|
| 76 |
+
- **LoRA Adapter**: Smaller adapter files (`adapter_model.safetensors`, `adapter_config.json`)
|
| 77 |
+
- **Tokenizer**: Shared tokenizer files for both options
|
| 78 |
+
|
| 79 |
## Technical Details
|
| 80 |
|
| 81 |
- **Base Model**: teknium/OpenHermes-2.5-Mistral-7B
|