ModernBERT fine-tuning complete

README.md

@@ -0,0 +1,67 @@
+---
+library_name: transformers
+license: apache-2.0
+base_model: distilbert-base-uncased
+tags:
+- generated_from_trainer
+metrics:
+- accuracy
+- f1
+model-index:
+- name: distilbert-web3-classification
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+# distilbert-web3-classification
+
+This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on the None dataset.
+It achieves the following results on the evaluation set:
+- Loss: 1.2908
+- Accuracy: 0.6672
+- F1: 0.6550
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 2e-05
+- train_batch_size: 32
+- eval_batch_size: 64
+- seed: 42
+- optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
+- lr_scheduler_type: linear
+- num_epochs: 20
+
+### Training results
+
+| Training Loss | Epoch | Step | Validation Loss | Accuracy | F1     |
+|:-------------:|:-----:|:----:|:---------------:|:--------:|:------:|
+| 1.4162        | 1.0   | 1361 | 1.3294          | 0.5749   | 0.5271 |
+| 1.1807        | 2.0   | 2722 | 1.2292          | 0.6163   | 0.5789 |
+| 0.9574        | 3.0   | 4083 | 1.1857          | 0.6430   | 0.6207 |
+| 0.7361        | 4.0   | 5444 | 1.1896          | 0.6688   | 0.6510 |
+| 0.5548        | 5.0   | 6805 | 1.2908          | 0.6672   | 0.6550 |
+
+
+### Framework versions
+
+- Transformers 4.50.3
+- Pytorch 2.6.0+cu124
+- Datasets 3.5.0
+- Tokenizers 0.21.1

config.json

@@ -0,0 +1,82 @@
+{
+  "activation": "gelu",
+  "architectures": [
+    "DistilBertForSequenceClassification"
+  ],
+  "attention_dropout": 0.1,
+  "dim": 768,
+  "dropout": 0.1,
+  "hidden_dim": 3072,
+  "id2label": {
+    "0": "unknown",
+    "1": "ai-agent",
+    "2": "desci",
+    "3": "depin",
+    "4": "memefi",
+    "5": "runes",
+    "6": "erc314",
+    "7": "dn404",
+    "8": "erc404",
+    "9": "rwa",
+    "10": "ordinals",
+    "11": "inscriptions",
+    "12": "ai",
+    "13": "socialfi",
+    "14": "tool",
+    "15": "gamblefi",
+    "16": "memecoin",
+    "17": "automation",
+    "18": "stablecoin",
+    "19": "other",
+    "20": "infrastructure",
+    "21": "metaverse",
+    "22": "dao",
+    "23": "gamefi",
+    "24": "nftfi",
+    "25": "defi",
+    "26": "nft"
+  },
+  "initializer_range": 0.02,
+  "label2id": {
+    "ai": 12,
+    "ai-agent": 1,
+    "automation": 17,
+    "dao": 22,
+    "defi": 25,
+    "depin": 3,
+    "desci": 2,
+    "dn404": 7,
+    "erc314": 6,
+    "erc404": 8,
+    "gamblefi": 15,
+    "gamefi": 23,
+    "infrastructure": 20,
+    "inscriptions": 11,
+    "memecoin": 16,
+    "memefi": 4,
+    "metaverse": 21,
+    "nft": 26,
+    "nftfi": 24,
+    "ordinals": 10,
+    "other": 19,
+    "runes": 5,
+    "rwa": 9,
+    "socialfi": 13,
+    "stablecoin": 18,
+    "tool": 14,
+    "unknown": 0
+  },
+  "max_position_embeddings": 512,
+  "model_type": "distilbert",
+  "n_heads": 12,
+  "n_layers": 6,
+  "pad_token_id": 0,
+  "problem_type": "single_label_classification",
+  "qa_dropout": 0.1,
+  "seq_classif_dropout": 0.2,
+  "sinusoidal_pos_embds": false,
+  "tie_weights_": true,
+  "torch_dtype": "float32",
+  "transformers_version": "4.50.3",
+  "vocab_size": 30522
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ea29104999649995888016a4be1340293d45f31251946171e186c83c71e49ad7
+size 267909468

runs/Mar30_21-22-42_fcbf7f43ac4f/events.out.tfevents.1743369765.fcbf7f43ac4f.4169.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1badff0ae3252f41013f12a80baabe9febf02aa651eb4537e79b76297c17eb40
+size 6054

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2b15af805054def9d4954a03589928b2e9c8937bae7769ad633816c9d2a18591
+size 5368