| --- |
| license: mit |
| tags: |
| - text-classification |
| - crypto |
| - technology |
| - twitter |
| - x |
| - fasttext-distillation |
| --- |
| this is a classification model that sorts tweets/profiles off the probability that it is tech/crypto related. this was a model created for a job that fell short. this is a tf-idf model, distilled from a transformer model that I also made. maybe ill upload that soon |
|
|
| # Techpto Classifier |
|
|
| This repository contains a lightweight production classifier for detecting whether X/Twitter posts and profiles are crypto-related, tech-related, both, or neither. |
|
|
| ## Files |
|
|
| - `text_classifier.json`: Rust-compatible hashed logistic-regression classifier. |
| - `model_config.json`: labels, expected inputs, and recommended thresholds. |
| - `distill_metrics.json`: proxy evaluation metrics from distillation. |
| - `recommended_thresholds_distillation.json`: thresholds tuned against the V7 fastText teacher. |
| - `full_run_manifest.json`: counts and thresholds from the large full-corpus run. |
|
|
| ## Recommended Thresholds |
|
|
| The high-precision full-corpus run used: |
|
|
| ```json |
| { |
| "post_crypto": 0.85, |
| "post_tech": 0.90, |
| "profile_crypto": 0.90, |
| "profile_tech": 0.99 |
| } |
| ``` |
|
|
| The original distillation-tuned thresholds were: |
|
|
| ```json |
| { |
| "post_crypto": 0.58, |
| "post_tech": 0.44, |
| "profile_crypto": 0.34, |
| "profile_tech": 0.38 |
| } |
| ``` |
|
|
| ## Full-Corpus Run |
|
|
| Using the high-precision thresholds: |
|
|
| - Posts scanned: `928,484,069` |
| - Post matches: `7,728,133` |
| - Profiles scanned: `2,667,815,773` |
| - Profile matches: `7,915,096` |
|
|
| One corrupt post shard was skipped and is listed in `full_run_manifest.json`. |
|
|
| ## Notes |
|
|
| This is not a standard Transformers checkpoint. It is a compact hashed-feature linear classifier intended for very high-throughput local scanning. Metrics in `distill_metrics.json` are proxy metrics against teacher/weak labels rather than a final human-labeled benchmark. |
|
|
