Instructions to use jfrick/scholarly-code-tokenizer with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use jfrick/scholarly-code-tokenizer with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("jfrick/scholarly-code-tokenizer", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Download research/modal-smoke.json from jfrick/scholarly-code-tokenizer: direct link, hf CLI and curl.
- Browser
- Download file 944 Bytes
-
https://huggingface.co/jfrick/scholarly-code-tokenizer/resolve/main/research/modal-smoke.json
- Command line
-
hf download hf://jfrick/scholarly-code-tokenizer/research/modal-smoke.json
-
curl -L -o modal-smoke.json https://huggingface.co/jfrick/scholarly-code-tokenizer/resolve/main/research/modal-smoke.json
944 Bytes
| { | |
| "returncode": 0, | |
| "stdout": "", | |
| "stderr": "test_corruption_and_unrequested_candidate_rejected (test_tokenizer_research.LearningCurveTests.test_corruption_and_unrequested_candidate_rejected) ... ok\ntest_cross_split_duplicate_quarantine (test_tokenizer_research.LearningCurveTests.test_cross_split_duplicate_quarantine) ... ok\ntest_interrupted_uncommitted_batch_replays_from_durable_cursor (test_tokenizer_research.LearningCurveTests.test_interrupted_uncommitted_batch_replays_from_durable_cursor) ... ok\ntest_nested_utf8_samples_and_shared_evaluation (test_tokenizer_research.LearningCurveTests.test_nested_utf8_samples_and_shared_evaluation) ... ok\ntest_resume_and_identity_guard (test_tokenizer_research.LearningCurveTests.test_resume_and_identity_guard) ... ok\n\n----------------------------------------------------------------------\nRan 5 tests in 0.915s\n\nOK\n", | |
| "execution": "modal-cpu", | |
| "cpu": 2, | |
| "memory_mib": 4096 | |
| } | |