Instructions to use jfrick/scholarly-code-tokenizer with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use jfrick/scholarly-code-tokenizer with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("jfrick/scholarly-code-tokenizer", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Download research/execution-history.json from jfrick/scholarly-code-tokenizer: direct link, hf CLI and curl.
- Browser
- Download file 2.86 kB
-
https://huggingface.co/jfrick/scholarly-code-tokenizer/resolve/main/research/execution-history.json
- Command line
-
hf download hf://jfrick/scholarly-code-tokenizer/research/execution-history.json
-
curl -L -o execution-history.json https://huggingface.co/jfrick/scholarly-code-tokenizer/resolve/main/research/execution-history.json
2.86 kB
| { | |
| "date": "2026-10-16", | |
| "active_run_id": "learning-curve-20261016-v2", | |
| "status": "completed", | |
| "modal_app_url": "https://modal.com/apps/radiantai/main/ap-A86MK9NsJAHwye2lcVuxyv", | |
| "local_tests_passed": 21, | |
| "remote_research_tests_passed": 5, | |
| "attempts": [ | |
| { | |
| "app_id": "ap-zrRin23bx0E5h11kA5Pxec", | |
| "outcome": "image_configuration_failed_before_worker_execution", | |
| "resolution": "Moved image environment configuration before add_local_* mounts." | |
| }, | |
| { | |
| "app_id": "ap-RkN0bYeRHWNVTdndcNZ7pR", | |
| "outcome": "remote_smoke_failed_before_collection", | |
| "resolution": "Resolved adapter source hash from the imported function rather than a checkout-specific filesystem path." | |
| }, | |
| { | |
| "app_id": "ap-5NdqpPYSaOsIIAbQdHc80V", | |
| "outcome": "remote_tests_passed_collection_stopped_for_throughput_improvement", | |
| "resolution": "Batched transactional writes; preserved partial v1 corpus and started a fresh v2 version. No tokenizer candidates fitted in this attempt." | |
| }, | |
| { | |
| "app_id": "ap-GSsPP18jVU2lZFhFnYy8Vw", | |
| "outcome": "collection_checkpoint_preserved_then_resumed" | |
| }, | |
| { | |
| "app_id": "ap-A86MK9NsJAHwye2lcVuxyv", | |
| "outcome": "corpus_frozen_all_18_candidates_completed", | |
| "resolution": "Upload checkpoints at one-minute intervals; same sampling code, spec and corpus identity." | |
| }, | |
| { | |
| "app_id": "ap-S2OL0auRAcvbx1kCqH7Njd", | |
| "outcome": "evaluation_8k_parent_cap_exhausted_libretexts_inventory", | |
| "resolution": "Preserved failed evaluation staging; fresh balanced-evaluation-v2 uses a 16 KiB parent/document cap and unchanged 32 MiB source quotas. No tokenizer was retrained." | |
| }, | |
| { | |
| "app_id": "ap-F9Q0Rx6GiCDPRcyHzGCaFo", | |
| "outcome": "broader_evaluation_frozen", | |
| "evaluation_bytes": 33554432, | |
| "source_scoped_parents": 4673, | |
| "libretexts_parents": 67 | |
| }, | |
| { | |
| "app_id": "ap-gOCdUYX2WO6VMVzaDuLZVX", | |
| "outcome": "all_18_candidates_rescored_on_shared_broader_evaluation", | |
| "correctness_passed": 18 | |
| } | |
| ], | |
| "authorization": "User explicitly authorized Modal measurements and removed the restored-ledger prerequisite. Resource and time bounds remain enforced.", | |
| "remote_test_app_url": "https://modal.com/apps/radiantai/main/ap-CrETuNnj4IkJF8aPh4mqmq", | |
| "evaluation_audit": { | |
| "original_libretexts_parents": 5, | |
| "initial_broader_cap_bytes": 8192, | |
| "libretexts_bytes_available_at_8k_cap": 636017, | |
| "final_broader_cap_bytes": 16384, | |
| "decision_basis": "Parent-coverage audit before interpretation of candidate learning curves." | |
| }, | |
| "fitted_candidates": 18, | |
| "rescored_candidates": 18, | |
| "evaluation_app_url": "https://modal.com/apps/radiantai/main/ap-gOCdUYX2WO6VMVzaDuLZVX", | |
| "sum_fitting_seconds": 927.888957187, | |
| "max_peak_process_rss_mib": 3531.80859375 | |
| } | |