File size: 2,391 Bytes
e02dee9 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 | # Lightweight LLM with MCP Integration
This directory contains a lightweight LLM inference service that can connect to external MCP (Model Context Protocol) servers for enhanced capabilities.
## 🏗️ Quick Setup
### Local Development
```bash
# Install dependencies
pip install -r requirements.txt
# Run the service
python app.py
# Or use the start script
./start.sh
```
### Docker Deployment
```bash
# Build the image
docker build -t lightweight-llm .
# Run the container
docker run -p 8000:8000 \
-e MCP_SERVER_URL=https://your-mcp-server.hf.space \
lightweight-llm
```
### Hugging Face Spaces
1. Create a new Space with Docker SDK
2. Upload all files from this directory
3. Set `app_port: 8000` in README.md header
4. Configure environment variables in Space settings
## 📡 API Usage
### Basic Inference
```bash
curl -X POST http://localhost:8000/ \
-H "Content-Type: application/json" \
-d '{
"inputs": "Hello, how are you?",
"parameters": {"max_new_tokens": 100}
}'
```
### MCP-Enhanced Inference
```bash
curl -X POST http://localhost:8000/ \
-H "Content-Type: application/json" \
-d '{
"inputs": "Tell me about product 123",
"parameters": {
"max_new_tokens": 200,
"mcp_server_url": "https://your-mcp-server.hf.space"
}
}'
```
## 🔧 Configuration
Set these environment variables:
- `MCP_SERVER_URL`: Default MCP server URL
- `MODEL_NAME`: Hugging Face model name
- `MAX_NEW_TOKENS`: Default max tokens
- `PORT`: Service port (default: 8000)
## 🧪 Testing
Run the test suite:
```bash
python test_service.py
```
## 🚀 Production Notes
- The current implementation uses a simple text generation placeholder
- For production, integrate with:
- Transformers library with a small model
- vLLM for faster inference
- TensorRT for optimized inference
- Any other inference engine
## 🔗 MCP Integration
This service automatically detects when to use MCP tools based on keywords in the input. It can connect to any MCP server that implements the standard protocol.
**Keywords that trigger MCP usage:**
- product, price, stock, inventory
- order, customer, database
- search, find, get, fetch, check
- prestashop, shop, cart, purchase
## 📊 Endpoints
- `POST /` - Main inference endpoint (HF compatible)
- `GET /health` - Health check
- `GET /info` - Service information
- `GET /docs` - API documentation
|