Upload README.md with huggingface_hub

README.md

@@ -20,38 +20,28 @@ This is a fine-tuned LoRA adapter for `fastino/gliner2-large-v1` trained to extr
 
 ---
 
-## Model Schema & Label Prefix
-
-The model is trained on a **7-field all-string schema** to optimize the GLiNER2 count head and prevent collapse. 
+## How to Get Started with the Model
 
-### Label Prefix
-Every input text **must** be prepended with the following fixed label prefix:
-```
-specificity: named | descriptive | vague usage: primary | supporting | background |
-```
+It is **highly recommended** to use this model through the official **`ai4data`** Python library wrapper. The library automatically handles:
+- **Markdown-aware chunking** (respecting model context limits).
+- **Character offset index adjustment** across multiple chunk pages.
+- **Greedy overlap resolution** and text normalization.
+- **Pre-filtering pre-classifiers** to skip non-data pages.
+- **Deduplication** and acronym matching.
 
-### Schema Structure
-```python
-SCHEMA = {
-    "data_mention": [
-        "name::str::The exact full name of the data source or dataset",
-        "acronym::str::The acronym or abbreviation if any",
-        "specificity::str::Whether this mention is named, descriptive, or vague",
-        "usage::str::Whether this is primary, supporting, or background data",
-        "datatype::str::The type of data verbatim from text such as survey, report, census, program, system, or assessment",
-        "producer::str::The organization or entity that produced or published the data",
-        "timeframe::str::The year or time period of the data such as 2019 or 2019 to 2020",
-    ]
-}
-```
+### 1. Installation
 
----
+Clone and install the repository:
 
-## How to Get Started with the Model
+```bash
+git clone <repository-url>
+cd monitoring_of_datause
+uv sync
+```
 
-### 1. Using the `ai4data` Python Library (Recommended)
+### 2. Python Usage
 
-If you are using the repository's native wrapper library, simply import and run:
+To extract dataset mentions and their attributes (like timeframe, producer, and acronyms):
 
 ```python
 from ai4data import extract_from_text, extract_from_document
@@ -69,57 +59,73 @@ pdf_results = extract_from_document("report.pdf", pages=[0, 1, 2])
 print(pdf_results)
 ```
 
-### 2. Direct Usage via standard `gliner2` Library
-
-If you want to use the raw `gliner2` model and adapter:
+---
 
-```python
-from gliner2 import GLiNER2
-from huggingface_hub import snapshot_download
-
-# Load the base model and LoRA adapter
-model = GLiNER2.from_pretrained("fastino/gliner2-large-v1")
-adapter_path = snapshot_download("ai4data/datause-extraction-v2")
-model.load_adapter(adapter_path)
-model.eval()
-
-# Configure the entity and relation schemas
-schema = model.create_schema()
-schema.entities({
-    "name": "The exact full name of the data source or dataset",
-    "acronym": "The acronym or abbreviation if any",
-    "producer": "The organization or entity that produced the data",
-    "timeframe": "The year or time period such as 2019 or 2019 to 2020",
-    "datatype": "The type of data verbatim from text",
-    "specificity": "Whether this mention is named, descriptive, or vague",
-    "usage": "Whether this is primary, supporting, or background data",
-})
-schema.relations({
-    "has_acronym": "The acronym of the dataset",
-    "has_producer": "The producer of the dataset",
-    "has_timeframe": "The timeframe of the dataset",
-    "has_datatype": "The data type of the dataset",
-    "has_specificity": "Whether this dataset is named, descriptive, or vague",
-    "has_usage": "Whether this dataset is primary, supporting, or background",
-})
-
-# format input text with the required label prefix
-text = "We use the Ghana Living Standard Survey (GLSS) 2020 conducted by Ghana Statistical Service."
-prefix = "specificity: named | descriptive | vague usage: primary | supporting | background |"
-prefixed_text = f"{prefix} {text}"
-
-# Extract
-result = model.extract(
-    prefixed_text,
-    schema,
-    threshold=0.3,
-    include_confidence=True,
-    include_spans=True,
-)
-
-print(result)
+## Model Schema & Response Structure
+
+The model extracts up to 7 attributes per data mention. When querying via `ai4data`, each extracted entity in the `"datasets"` list has the following structure:
+
+```json
+{
+  "mention_name": {
+    "text": "Demographic and Health Survey",
+    "confidence": 0.9999,
+    "start": 23,
+    "end": 52
+  },
+  "specificity_tag": {
+    "text": "named",
+    "confidence": 0.9999,
+    "start": 23,
+    "end": 52
+  },
+  "usage_context": {
+    "text": "primary",
+    "confidence": 0.9999,
+    "start": 23,
+    "end": 52
+  },
+  "typology_tag": {
+    "text": "survey",
+    "confidence": 0.9999,
+    "start": 23,
+    "end": 52
+  },
+  "acronym": {
+    "text": "DHS",
+    "confidence": 0.9996,
+    "start": 54,
+    "end": 57
+  },
+  "producer": {
+    "text": "National Statistics Office",
+    "confidence": 0.9999,
+    "start": 72,
+    "end": 98
+  },
+  "reference_year": {
+    "text": "2022",
+    "confidence": 0.9999,
+    "start": 18,
+    "end": 22
+  },
+  "is_used": {
+    "text": "True",
+    "confidence": 0.9999,
+    "start": 23,
+    "end": 52
+  },
+  "geography": {
+    "text": "",
+    "confidence": 0.9999,
+    "start": 23,
+    "end": 52
+  }
+}
 ```
 
+---
+
 ## Annotation Guidelines & What Counts as a Data Mention
 
 ### Specificity Taxonomy