Update README.md
Browse files
README.md
CHANGED
|
@@ -202,7 +202,7 @@ for i, (c, t) in enumerate(zip(chunks, token_pos)):
|
|
| 202 |
print(f"-----chunk: {i}----token_idx: {t}--------")
|
| 203 |
print(c)
|
| 204 |
```
|
| 205 |
-
##
|
| 206 |
The following script supports specifying max tokens per chunk, which can be seen as a new experimental version of the scripts above.
|
| 207 |
```python
|
| 208 |
import torch
|
|
|
|
| 202 |
print(f"-----chunk: {i}----token_idx: {t}--------")
|
| 203 |
print(c)
|
| 204 |
```
|
| 205 |
+
## Experimental
|
| 206 |
The following script supports specifying max tokens per chunk, which can be seen as a new experimental version of the scripts above.
|
| 207 |
```python
|
| 208 |
import torch
|