README.md

---
title: Tokenizer
emoji: 📚
colorFrom: green
colorTo: yellow
sdk: docker
pinned: false
---

# Tokenizer API

A simple API for tokenizing text using spaCy.

## Usage Examples

### Using curl
```bash
curl -X POST "https://theirstory-tokenizer.hf.space/tokenize" \
     -H "Content-Type: application/json" \
     -H "accept: application/json" \
     -d '{"text": "Hello world"}'
```

### Using Python
```python
import requests

response = requests.post(
    "https://theirstory-tokenizer.hf.space/tokenize",
    json={"text": "Hello world"},
    headers={"accept": "application/json"}
)
tokens = response.json()
print(tokens)
```

### Using JavaScript
```javascript
const response = await fetch('https://theirstory-tokenizer.hf.space/tokenize', {
    method: 'POST',
    headers: {
        'Content-Type': 'application/json',
        'accept': 'application/json'
    },
    body: JSON.stringify({
        text: 'Hello world'
    })
});
const tokens = await response.json();
console.log(tokens);
```

Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference