Instructions to use jfrick/scholarly-code-tokenizer with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use jfrick/scholarly-code-tokenizer with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("jfrick/scholarly-code-tokenizer", device_map="auto") - Notebooks
- Google Colab
- Kaggle
File size: 2,856 Bytes
6bbbc6f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 | {
"date": "2026-10-16",
"active_run_id": "learning-curve-20261016-v2",
"status": "completed",
"modal_app_url": "https://modal.com/apps/radiantai/main/ap-A86MK9NsJAHwye2lcVuxyv",
"local_tests_passed": 21,
"remote_research_tests_passed": 5,
"attempts": [
{
"app_id": "ap-zrRin23bx0E5h11kA5Pxec",
"outcome": "image_configuration_failed_before_worker_execution",
"resolution": "Moved image environment configuration before add_local_* mounts."
},
{
"app_id": "ap-RkN0bYeRHWNVTdndcNZ7pR",
"outcome": "remote_smoke_failed_before_collection",
"resolution": "Resolved adapter source hash from the imported function rather than a checkout-specific filesystem path."
},
{
"app_id": "ap-5NdqpPYSaOsIIAbQdHc80V",
"outcome": "remote_tests_passed_collection_stopped_for_throughput_improvement",
"resolution": "Batched transactional writes; preserved partial v1 corpus and started a fresh v2 version. No tokenizer candidates fitted in this attempt."
},
{
"app_id": "ap-GSsPP18jVU2lZFhFnYy8Vw",
"outcome": "collection_checkpoint_preserved_then_resumed"
},
{
"app_id": "ap-A86MK9NsJAHwye2lcVuxyv",
"outcome": "corpus_frozen_all_18_candidates_completed",
"resolution": "Upload checkpoints at one-minute intervals; same sampling code, spec and corpus identity."
},
{
"app_id": "ap-S2OL0auRAcvbx1kCqH7Njd",
"outcome": "evaluation_8k_parent_cap_exhausted_libretexts_inventory",
"resolution": "Preserved failed evaluation staging; fresh balanced-evaluation-v2 uses a 16 KiB parent/document cap and unchanged 32 MiB source quotas. No tokenizer was retrained."
},
{
"app_id": "ap-F9Q0Rx6GiCDPRcyHzGCaFo",
"outcome": "broader_evaluation_frozen",
"evaluation_bytes": 33554432,
"source_scoped_parents": 4673,
"libretexts_parents": 67
},
{
"app_id": "ap-gOCdUYX2WO6VMVzaDuLZVX",
"outcome": "all_18_candidates_rescored_on_shared_broader_evaluation",
"correctness_passed": 18
}
],
"authorization": "User explicitly authorized Modal measurements and removed the restored-ledger prerequisite. Resource and time bounds remain enforced.",
"remote_test_app_url": "https://modal.com/apps/radiantai/main/ap-CrETuNnj4IkJF8aPh4mqmq",
"evaluation_audit": {
"original_libretexts_parents": 5,
"initial_broader_cap_bytes": 8192,
"libretexts_bytes_available_at_8k_cap": 636017,
"final_broader_cap_bytes": 16384,
"decision_basis": "Parent-coverage audit before interpretation of candidate learning curves."
},
"fitted_candidates": 18,
"rescored_candidates": 18,
"evaluation_app_url": "https://modal.com/apps/radiantai/main/ap-gOCdUYX2WO6VMVzaDuLZVX",
"sum_fitting_seconds": 927.888957187,
"max_peak_process_rss_mib": 3531.80859375
}
|