Cursor Agent commited on
Commit
a930d30
·
1 Parent(s): ffe15e5

fix: use huggingface-hub 0.17.3 for tokenizers compatibility

Browse files

CRITICAL FIX for tokenizers dependency:
- tokenizers 0.14.x requires huggingface-hub<0.18
- Previous 0.19.4 was too new
- Now using 0.17.3 (highest compatible version)

Version constraints satisfied:
- transformers 4.35.0: ✅ (needs >=0.16.4,<1.0)
- datasets 2.14.5: ✅ (needs >=0.14.0,<1.0)
- tokenizers 0.14.x: ✅ (needs >=0.16.4,<0.18)

This should resolve immediately without backtracking.

Files changed (1) hide show
  1. requirements.txt +6 -3
requirements.txt CHANGED
@@ -37,9 +37,12 @@ watchdog==6.0.0
37
  dnspython==2.7.0
38
 
39
  # HuggingFace (optional - for AI models)
40
- # PINNED VERSIONS to avoid pip backtracking hell
41
- # These versions are tested and compatible
42
- huggingface-hub==0.19.4
 
 
 
43
  datasets==2.14.5
44
 
45
  # Utilities
 
37
  dnspython==2.7.0
38
 
39
  # HuggingFace (optional - for AI models)
40
+ # CRITICAL VERSION CONSTRAINTS:
41
+ # - transformers 4.35.0 requires: huggingface-hub>=0.16.4,<1.0
42
+ # - datasets 2.14.5 requires: huggingface-hub>=0.14.0,<1.0.0
43
+ # - tokenizers 0.14.x requires: huggingface-hub>=0.16.4,<0.18 ← THE CONSTRAINT!
44
+ # Compatible version: 0.16.4 <= huggingface-hub < 0.18
45
+ huggingface-hub==0.17.3 # Maximum version compatible with tokenizers
46
  datasets==2.14.5
47
 
48
  # Utilities