improved readme

README.md

@@ -8,198 +8,102 @@ metrics:
 base_model:
 - yiyanghkust/finbert-pretrain
 library_name: transformers
+tags:
+- finance
+- ai-detector
+- sequence-classification
 ---
-# Model Card for Model ID
 
-<!-- Provide a quick summary of what the model is/does. -->
+# FinBERT AI Detector for Financial Texts
 
-This modelcard aims to be a base template for new models. It has been generated using [this raw template](https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/templates/modelcard_template.md?plain=1).
+This model is a fine-tuned version of `yiyanghkust/finbert-pretrain` designed specifically to detect AI-generated text in financial documents, such as corporate annual reports (e.g., 10-K filings).
 
 ## Model Details
 
-### Model Description
+- **Base Model:** `yiyanghkust/finbert-pretrain`
+- **Task:** Sequence Classification (Binary: Human vs. AI-Generated)
+- **Language:** English
+- **License:** MIT
 
-<!-- Provide a longer summary of what this model is. -->
+## Training Data and Method
 
+The model was trained on a custom dataset compiled from human-written financial texts (derived from SEC annual reports) and AI-generated equivalents.
 
-
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-
-### Model Sources [optional]
-
-<!-- Provide the basic links for the model. -->
-
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
+- **Data Generation:** Actual human texts from corporate annual reports were compiled. State-of-the-art Large Language Models (LLMs), including OpenAI's ChatGPT, Google's Gemini, and Anthropic's Claude, were then prompted to rewrite these sections or generate similar artificial financial texts.
+- **Training Method:** The base `finbert-pretrain` model—already pre-trained on a large corpus of financial text—was fine-tuned on this mixed dataset to classify whether a given segment of text is human-written or generated by an AI.
 
 ## Uses
 
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-
-### Direct Use
-
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-[More Information Needed]
-
-### Downstream Use [optional]
-
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-
-[More Information Needed]
-
-### Out-of-Scope Use
-
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-
-[More Information Needed]
-
-## Bias, Risks, and Limitations
-
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-
-[More Information Needed]
-
-### Recommendations
-
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+This model is intended for researchers, financial analysts, and auditors who want to verify the authenticity of corporate disclosures and determine if a financial text (like an annual report or an earnings call transcript) was written by an AI or a human.
 
 ## How to Get Started with the Model
 
-Use the code below to get started with the model.
-
-[More Information Needed]
-
-## Training Details
-
-### Training Data
-
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-
-[More Information Needed]
-
-### Training Procedure
-
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-
-#### Preprocessing [optional]
-
-[More Information Needed]
-
-
-#### Training Hyperparameters
-
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-
-#### Speeds, Sizes, Times [optional]
-
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-
-[More Information Needed]
-
-## Evaluation
-
-<!-- This section describes the evaluation protocols and provides the results. -->
-
-### Testing Data, Factors & Metrics
-
-#### Testing Data
-
-<!-- This should link to a Dataset Card if possible. -->
-
-[More Information Needed]
-
-#### Factors
-
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
-
-#### Metrics
-
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
-
-### Results
-
-[More Information Needed]
-
-#### Summary
-
-
-
-## Model Examination [optional]
-
-<!-- Relevant interpretability work for the model goes here -->
-
-[More Information Needed]
-
-## Environmental Impact
-
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-
-## Technical Specifications [optional]
-
-### Model Architecture and Objective
-
-[More Information Needed]
-
-### Compute Infrastructure
-
-[More Information Needed]
-
-#### Hardware
-
-[More Information Needed]
-
-#### Software
+You can use this model with the `transformers` library. Here is a quick example of how to make predictions using Python and PyTorch:
 
-[More Information Needed]
+```python
+import torch
+from transformers import AutoTokenizer, AutoModelForSequenceClassification
 
-## Citation [optional]
+# Load the model and tokenizer (replace 'username/model-name' with your Hugging Face model ID)
+model_id = "your-username/your-model-name"
+device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
 
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForSequenceClassification.from_pretrained(model_id).to(device)
+model.eval()
 
-**BibTeX:**
+# Sample text to test
+text = "The Tax Cuts and Jobs Act enacted in 2017 in the United States, significantly changed the tax rules applicable to U.S.-domiciled corporations. Changes such as lower corporate tax rates, full expensing for qualified property, taxation of offshore earnings, limitations on interest expense deductions, and changes to the municipal bond tax exemption may impact demand for our products and services."
 
-[More Information Needed]
+# Preprocess the input
+inputs = tokenizer(
+    text,
+    return_tensors="pt",
+    truncation=True,
+    max_length=512,
+    padding=True
+).to(device)
 
-**APA:**
+# Run inference
+with torch.no_grad():
+    outputs = model(**inputs)
+    probabilities = torch.nn.functional.softmax(outputs.logits, dim=-1)
 
-[More Information Needed]
+# Index 1 typically corresponds to "AI-generated" (Verify with model.config.id2label if needed)
+prob_ai = probabilities[0][1].item()
 
-## Glossary [optional]
+print(f"Probability of being AI-generated: {prob_ai:.2%}")
+```
 
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+### Batch Processing Example
 
-[More Information Needed]
+For larger text datasets, you can generate predictions in batches to properly utilize GPU acceleration:
 
-## More Information [optional]
+```python
+texts = [
+    "Company revenue grew by 15% due to increased demand in the European market.",
+    "A machine learning model generated this text based on recent financial statements."
+]
 
-[More Information Needed]
+inputs = tokenizer(
+    texts,
+    return_tensors="pt",
+    truncation=True,
+    max_length=512,
+    padding=True
+).to(device)
 
-## Model Card Authors [optional]
+with torch.no_grad():
+    outputs = model(**inputs)
+    probabilities = torch.nn.functional.softmax(outputs.logits, dim=-1)
 
-[More Information Needed]
+probs_ai = probabilities[:, 1].tolist()
+for text, prob in zip(texts, probs_ai):
+    print(f"AI Probability: {prob:.2%} - Text: {text}")
+```
 
-## Model Card Contact
+## Intended Use & Limitations
 
-[More Information Needed]
+- **Intended Usage:** Analyzing formal financial reports, press releases, corporate filings, and similar structured financial disclosures.
+- **Limitations:** The model is optimized specifically for the formal, complex tone of financial documents. Its accuracy may be lower when applied to texts outside the financial domain, such as social media posts, casual emails, news articles, or creative text.
+- **Length Constraint:** The underlying standard FinBERT architecture implies a maximum sequence length of 512 tokens. Texts longer than this will be truncated.

colab-notebooks/testing-fin-ai-detector.ipynb

@@ -1 +1,608 @@
-{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"provenance":[],"authorship_tag":"ABX9TyMlpd3OM9jBfwFJVtp5BDWy"},"kernelspec":{"name":"python3","display_name":"Python 3"},"language_info":{"name":"python"},"widgets":{"application/vnd.jupyter.widget-state+json":{"0bcf00c2850b4166b19fa2cc67b2c316":{"model_module":"@jupyter-widgets/controls","model_name":"HBoxModel","model_module_version":"1.5.0","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HBoxModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HBoxView","box_style":"","children":["IPY_MODEL_f9fc8866d45d4488849542210285a0f6","IPY_MODEL_ac892e73e2f146eb93ae1501a88bd42a","IPY_MODEL_68f442f24a8d4009a7dc629bd8e523ec"],"layout":"IPY_MODEL_b006d255aefa465e9192dcf701587a77"}},"f9fc8866d45d4488849542210285a0f6":{"model_module":"@jupyter-widgets/controls","model_name":"HTMLModel","model_module_version":"1.5.0","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HTMLModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HTMLView","description":"","description_tooltip":null,"layout":"IPY_MODEL_57fd3787e2cd49cdbbb85d4fbfa289e4","placeholder":"​","style":"IPY_MODEL_db5a306e5c8042e89d63144789306505","value":"Loading weights: 100%"}},"ac892e73e2f146eb93ae1501a88bd42a":{"model_module":"@jupyter-widgets/controls","model_name":"FloatProgressModel","model_module_version":"1.5.0","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"FloatProgressModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"ProgressView","bar_style":"success","description":"","description_tooltip":null,"layout":"IPY_MODEL_18ddc56f9a3f4ca0bf9f9234ddc1cf43","max":201,"min":0,"orientation":"horizontal","style":"IPY_MODEL_68b3487aabe245c7a868dd04ca88da73","value":201}},"68f442f24a8d4009a7dc629bd8e523ec":{"model_module":"@jupyter-widgets/controls","model_name":"HTMLModel","model_module_version":"1.5.0","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HTMLModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HTMLView","description":"","description_tooltip":null,"layout":"IPY_MODEL_67061d2557144f488710cebd793bb96b","placeholder":"​","style":"IPY_MODEL_f3e0fbc3cf534e3ea70089599ccbde10","value":" 201/201 [00:00&lt;00:00, 513.14it/s, Materializing param=classifier.weight]"}},"b006d255aefa465e9192dcf701587a77":{"model_module":"@jupyter-widgets/base","model_name":"LayoutModel","model_module_version":"1.2.0","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"57fd3787e2cd49cdbbb85d4fbfa289e4":{"model_module":"@jupyter-widgets/base","model_name":"LayoutModel","model_module_version":"1.2.0","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"db5a306e5c8042e89d63144789306505":{"model_module":"@jupyter-widgets/controls","model_name":"DescriptionStyleModel","model_module_version":"1.5.0","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"DescriptionStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","description_width":""}},"18ddc56f9a3f4ca0bf9f9234ddc1cf43":{"model_module":"@jupyter-widgets/base","model_name":"LayoutModel","model_module_version":"1.2.0","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"68b3487aabe245c7a868dd04ca88da73":{"model_module":"@jupyter-widgets/controls","model_name":"ProgressStyleModel","model_module_version":"1.5.0","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"ProgressStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","bar_color":null,"description_width":""}},"67061d2557144f488710cebd793bb96b":{"model_module":"@jupyter-widgets/base","model_name":"LayoutModel","model_module_version":"1.2.0","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"f3e0fbc3cf534e3ea70089599ccbde10":{"model_module":"@jupyter-widgets/controls","model_name":"DescriptionStyleModel","model_module_version":"1.5.0","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"DescriptionStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","description_width":""}}}}},"cells":[{"cell_type":"code","execution_count":55,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":49,"referenced_widgets":["0bcf00c2850b4166b19fa2cc67b2c316","f9fc8866d45d4488849542210285a0f6","ac892e73e2f146eb93ae1501a88bd42a","68f442f24a8d4009a7dc629bd8e523ec","b006d255aefa465e9192dcf701587a77","57fd3787e2cd49cdbbb85d4fbfa289e4","db5a306e5c8042e89d63144789306505","18ddc56f9a3f4ca0bf9f9234ddc1cf43","68b3487aabe245c7a868dd04ca88da73","67061d2557144f488710cebd793bb96b","f3e0fbc3cf534e3ea70089599ccbde10"]},"id":"DQnP_uKC0HZG","outputId":"89b3ed6b-011f-491d-e006-1aa5fc4335ab","executionInfo":{"status":"ok","timestamp":1774618900263,"user_tz":180,"elapsed":1301,"user":{"displayName":"Marcelo Perlin","userId":"18075542091515264462"}}},"outputs":[{"output_type":"display_data","data":{"text/plain":["Loading weights:   0%|          | 0/201 [00:00<?, ?it/s]"],"application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"0bcf00c2850b4166b19fa2cc67b2c316"}},"metadata":{}}],"source":["# Use a pipeline as a high-level helper\n","from transformers import pipeline\n","\n","pipe = pipeline(\"text-classification\", model=\"msperlin/finbert-ai-detector\")"]},{"cell_type":"code","source":["human_text = \"the last trading day of the immediately preceding calendar quarter is greater than or equal to 130% of the conversion price on each applicable trading day; (2) during the five business day period after any 10 consecutive trading day period (the measurement period) in which the trading price per $1,000 principal amount of notes for each trading day of the measurement period was less than 98% of the product of the last reported sale price of our common stock and the conversion rate on each such trading day; or (3) upon 51 the occurrence of specified corporate events. On or after June 1, 2026 and until the close of business on the business day immediately preceding the maturity date, holders may convert their notes at any time, regardless of the foregoing circumstances. Upon conversion, we will pay or deliver, as the case may be, cash, shares of our common stock or a combination of cash and shares of the our common stock, at our election. During the fourth quarter of 2017, none of the conversion conditions were satisfied and as a result, the 2.125% Notes are not eligible for conversion during the first calendar quarter of 2018.We may not redeem the notes prior to the maturity date, and no sinking fund is provided for the notes.\"\n","\n","ai_text= \"The conditions for conversion are met if: (1) the reported closing price of our common stock on the final trading day of the immediately preceding calendar quarter equals or exceeds 130% of the then-effective conversion price on each relevant trading day; (2) during the five business days immediately following any ten consecutive trading day period (hereinafter, the measurement period) wherein the daily trading price per $1,000 principal amount of notes throughout such measurement period was less than 98% of the product of the last reported sale price of our common stock and the conversion rate applicable on each such trading day; or (3) upon the occurrence of certain specified corporate events. Commencing June 1, 2026, and continuing through the close of business on the business day immediately preceding the maturity date, holders shall be entitled to convert their notes at any time, irrespective of the aforementioned conditions. Upon any such conversion, we retain the sole discretion to satisfy the conversion obligation by delivering cash, shares of our common stock, or a combination thereof. Throughout the fourth fiscal quarter of 2017, none of the requisite conversion conditions were met, thereby rendering the 2.125% Notes ineligible for conversion during the first calendar quarter of 2018. The notes are not subject to redemption by the Company prior to their stated maturity date, nor is any sinking fund provision established for them. The initial conversion rate is stipulated at 125.0031 shares of common stock per $1,000 principal amount of notes, corresponding to an initial conversion price of approximately $8.00 per share of common stock.\"\n","\n"],"metadata":{"id":"m-vN1Z6eXGLN","executionInfo":{"status":"ok","timestamp":1774618900269,"user_tz":180,"elapsed":3,"user":{"displayName":"Marcelo Perlin","userId":"18075542091515264462"}}},"execution_count":56,"outputs":[]},{"cell_type":"code","source":["import re\n","\n","def clean_financial_text(text):\n","    # 1. Standardize whitespace\n","    text = re.sub(r'\\\\s+', ' ', text)\n","\n","    # 2. Remove typical PDF conversion artifacts (e.g., Page X of Y)\n","    text = re.sub(r'Page \\d+ of \\d+', '', text, flags=re.IGNORECASE)\n","\n","    # 3. Keep common financial/punctuation non-ASCII, strip the rest\n","    # This regex keeps basic ASCII + common currency + accented Latin-1\n","    # Adjust if you're specifically looking for smart quotes as AI markers\n","    text = \"\".join(i for i in text if ord(i) < 128 or i in \"€£¥©®™áéíóúçÁÉÍÓÚÇ\")\n","\n","    # 20260326 dont converto to lower case\n","    #text = text.lower()\n","\n","    # 4. Strip leading/trailing whitespace\n","    text = text.strip()\n","\n","    return text"],"metadata":{"id":"a9flBHWxZzCW","executionInfo":{"status":"ok","timestamp":1774618900287,"user_tz":180,"elapsed":14,"user":{"displayName":"Marcelo Perlin","userId":"18075542091515264462"}}},"execution_count":57,"outputs":[]},{"cell_type":"code","source":["#human_text = clean_financial_text(human_text)\n","#ai_text = clean_financial_text(ai_text)\n","\n","res_human = pipe(human_text)\n","res_ai = pipe(ai_text)\n","\n","# Label 0 is human written, Label 1 is AI written\n","print(f\"Result for human text: {res_human}\")\n","print(f\"Result for AI text: {res_ai}\")"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"3QOYWfy3Xf2y","executionInfo":{"status":"ok","timestamp":1774618946943,"user_tz":180,"elapsed":1909,"user":{"displayName":"Marcelo Perlin","userId":"18075542091515264462"}},"outputId":"095ccc6b-28c3-4cac-ab05-1b29dd0e92c4"},"execution_count":60,"outputs":[{"output_type":"stream","name":"stdout","text":["Result for human text: [{'label': 'LABEL_0', 'score': 0.8389487862586975}]\n","Result for AI text: [{'label': 'LABEL_1', 'score': 0.9915664792060852}]\n"]}]}]}
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 49,
+     "referenced_widgets": [
+      "0bcf00c2850b4166b19fa2cc67b2c316",
+      "f9fc8866d45d4488849542210285a0f6",
+      "ac892e73e2f146eb93ae1501a88bd42a",
+      "68f442f24a8d4009a7dc629bd8e523ec",
+      "b006d255aefa465e9192dcf701587a77",
+      "57fd3787e2cd49cdbbb85d4fbfa289e4",
+      "db5a306e5c8042e89d63144789306505",
+      "18ddc56f9a3f4ca0bf9f9234ddc1cf43",
+      "68b3487aabe245c7a868dd04ca88da73",
+      "67061d2557144f488710cebd793bb96b",
+      "f3e0fbc3cf534e3ea70089599ccbde10"
+     ]
+    },
+    "executionInfo": {
+     "elapsed": 1301,
+     "status": "ok",
+     "timestamp": 1774618900263,
+     "user": {
+      "displayName": "Marcelo Perlin",
+      "userId": "18075542091515264462"
+     },
+     "user_tz": 180
+    },
+    "id": "DQnP_uKC0HZG",
+    "outputId": "89b3ed6b-011f-491d-e006-1aa5fc4335ab"
+   },
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/usr/local/lib/python3.12/dist-packages/huggingface_hub/utils/_auth.py:104: UserWarning: \n",
+      "Error while fetching `HF_TOKEN` secret value from your vault: 'Requesting secret HF_TOKEN timed out. Secrets can only be fetched when running from the Colab UI.'.\n",
+      "You are not authenticated with the Hugging Face Hub in this notebook.\n",
+      "If the error persists, please let us know by opening an issue on GitHub (https://github.com/huggingface/huggingface_hub/issues/new).\n",
+      "  warnings.warn(\n",
+      "Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.\n",
+      "WARNING:huggingface_hub.utils._http:Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.\n"
+     ]
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "8aaed08ad5424985ab9c188b43296373",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "config.json:   0%|          | 0.00/739 [00:00<?, ?B/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "ab6661749fb04bec9d00cd9ee9254cf1",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "model.safetensors:   0%|          | 0.00/439M [00:00<?, ?B/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "b4b6412d0cfa49b2a5aa5020246409f8",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "Loading weights:   0%|          | 0/201 [00:00<?, ?it/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "a7dcfe5b062b43e3a60f4138e63c0da6",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "tokenizer_config.json:   0%|          | 0.00/351 [00:00<?, ?B/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "ad960d7ceac74783bb5e1a449a6af2a3",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "tokenizer.json: 0.00B [00:00, ?B/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "# Use a pipeline as a high-level helper\n",
+    "from transformers import pipeline\n",
+    "\n",
+    "pipe = pipeline(\"text-classification\", model=\"msperlin/finbert-ai-detector\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {
+    "executionInfo": {
+     "elapsed": 3,
+     "status": "ok",
+     "timestamp": 1774618900269,
+     "user": {
+      "displayName": "Marcelo Perlin",
+      "userId": "18075542091515264462"
+     },
+     "user_tz": 180
+    },
+    "id": "m-vN1Z6eXGLN"
+   },
+   "outputs": [],
+   "source": [
+    "human_text = \"the last trading day of the immediately preceding calendar quarter is greater than or equal to 130% of the conversion price on each applicable trading day; (2) during the five business day period after any 10 consecutive trading day period (the measurement period) in which the trading price per $1,000 principal amount of notes for each trading day of the measurement period was less than 98% of the product of the last reported sale price of our common stock and the conversion rate on each such trading day; or (3) upon 51 the occurrence of specified corporate events. On or after June 1, 2026 and until the close of business on the business day immediately preceding the maturity date, holders may convert their notes at any time, regardless of the foregoing circumstances. Upon conversion, we will pay or deliver, as the case may be, cash, shares of our common stock or a combination of cash and shares of the our common stock, at our election. During the fourth quarter of 2017, none of the conversion conditions were satisfied and as a result, the 2.125% Notes are not eligible for conversion during the first calendar quarter of 2018.We may not redeem the notes prior to the maturity date, and no sinking fund is provided for the notes.\"\n",
+    "\n",
+    "ai_text= \"The conditions for conversion are met if: (1) the reported closing price of our common stock on the final trading day of the immediately preceding calendar quarter equals or exceeds 130% of the then-effective conversion price on each relevant trading day; (2) during the five business days immediately following any ten consecutive trading day period (hereinafter, the measurement period) wherein the daily trading price per $1,000 principal amount of notes throughout such measurement period was less than 98% of the product of the last reported sale price of our common stock and the conversion rate applicable on each such trading day; or (3) upon the occurrence of certain specified corporate events. Commencing June 1, 2026, and continuing through the close of business on the business day immediately preceding the maturity date, holders shall be entitled to convert their notes at any time, irrespective of the aforementioned conditions. Upon any such conversion, we retain the sole discretion to satisfy the conversion obligation by delivering cash, shares of our common stock, or a combination thereof. Throughout the fourth fiscal quarter of 2017, none of the requisite conversion conditions were met, thereby rendering the 2.125% Notes ineligible for conversion during the first calendar quarter of 2018. The notes are not subject to redemption by the Company prior to their stated maturity date, nor is any sinking fund provision established for them. The initial conversion rate is stipulated at 125.0031 shares of common stock per $1,000 principal amount of notes, corresponding to an initial conversion price of approximately $8.00 per share of common stock.\"\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {
+    "executionInfo": {
+     "elapsed": 14,
+     "status": "ok",
+     "timestamp": 1774618900287,
+     "user": {
+      "displayName": "Marcelo Perlin",
+      "userId": "18075542091515264462"
+     },
+     "user_tz": 180
+    },
+    "id": "a9flBHWxZzCW"
+   },
+   "outputs": [],
+   "source": [
+    "import re\n",
+    "\n",
+    "def clean_financial_text(text):\n",
+    "    # 1. Standardize whitespace\n",
+    "    text = re.sub(r'\\\\s+', ' ', text)\n",
+    "\n",
+    "    # 2. Remove typical PDF conversion artifacts (e.g., Page X of Y)\n",
+    "    text = re.sub(r'Page \\d+ of \\d+', '', text, flags=re.IGNORECASE)\n",
+    "\n",
+    "    # 3. Keep common financial/punctuation non-ASCII, strip the rest\n",
+    "    # This regex keeps basic ASCII + common currency + accented Latin-1\n",
+    "    # Adjust if you're specifically looking for smart quotes as AI markers\n",
+    "    text = \"\".join(i for i in text if ord(i) < 128 or i in \"€£¥©®™áéíóúçÁÉÍÓÚÇ\")\n",
+    "\n",
+    "    # 20260326 dont converto to lower case\n",
+    "    #text = text.lower()\n",
+    "\n",
+    "    # 4. Strip leading/trailing whitespace\n",
+    "    text = text.strip()\n",
+    "\n",
+    "    return text"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "executionInfo": {
+     "elapsed": 1909,
+     "status": "ok",
+     "timestamp": 1774618946943,
+     "user": {
+      "displayName": "Marcelo Perlin",
+      "userId": "18075542091515264462"
+     },
+     "user_tz": 180
+    },
+    "id": "3QOYWfy3Xf2y",
+    "outputId": "095ccc6b-28c3-4cac-ab05-1b29dd0e92c4"
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Result for human text: [{'label': 'LABEL_0', 'score': 0.8389487266540527}]\n",
+      "Result for AI text: [{'label': 'LABEL_1', 'score': 0.9915664792060852}]\n"
+     ]
+    }
+   ],
+   "source": [
+    "#human_text = clean_financial_text(human_text)\n",
+    "#ai_text = clean_financial_text(ai_text)\n",
+    "\n",
+    "res_human = pipe(human_text)\n",
+    "res_ai = pipe(ai_text)\n",
+    "\n",
+    "# Label 0 is human written, Label 1 is AI written\n",
+    "print(f\"Result for human text: {res_human}\")\n",
+    "print(f\"Result for AI text: {res_ai}\")"
+   ]
+  }
+ ],
+ "metadata": {
+  "colab": {
+   "authorship_tag": "ABX9TyMlpd3OM9jBfwFJVtp5BDWy",
+   "provenance": []
+  },
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.13"
+  },
+  "widgets": {
+   "application/vnd.jupyter.widget-state+json": {
+    "0bcf00c2850b4166b19fa2cc67b2c316": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "HBoxModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HBoxModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HBoxView",
+      "box_style": "",
+      "children": [
+       "IPY_MODEL_f9fc8866d45d4488849542210285a0f6",
+       "IPY_MODEL_ac892e73e2f146eb93ae1501a88bd42a",
+       "IPY_MODEL_68f442f24a8d4009a7dc629bd8e523ec"
+      ],
+      "layout": "IPY_MODEL_b006d255aefa465e9192dcf701587a77"
+     }
+    },
+    "18ddc56f9a3f4ca0bf9f9234ddc1cf43": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "57fd3787e2cd49cdbbb85d4fbfa289e4": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "67061d2557144f488710cebd793bb96b": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "68b3487aabe245c7a868dd04ca88da73": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "ProgressStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "ProgressStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "bar_color": null,
+      "description_width": ""
+     }
+    },
+    "68f442f24a8d4009a7dc629bd8e523ec": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "HTMLModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HTMLModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HTMLView",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_67061d2557144f488710cebd793bb96b",
+      "placeholder": "​",
+      "style": "IPY_MODEL_f3e0fbc3cf534e3ea70089599ccbde10",
+      "value": " 201/201 [00:00&lt;00:00, 513.14it/s, Materializing param=classifier.weight]"
+     }
+    },
+    "ac892e73e2f146eb93ae1501a88bd42a": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "FloatProgressModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "FloatProgressModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "ProgressView",
+      "bar_style": "success",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_18ddc56f9a3f4ca0bf9f9234ddc1cf43",
+      "max": 201,
+      "min": 0,
+      "orientation": "horizontal",
+      "style": "IPY_MODEL_68b3487aabe245c7a868dd04ca88da73",
+      "value": 201
+     }
+    },
+    "b006d255aefa465e9192dcf701587a77": {
+     "model_module": "@jupyter-widgets/base",
+     "model_module_version": "1.2.0",
+     "model_name": "LayoutModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/base",
+      "_model_module_version": "1.2.0",
+      "_model_name": "LayoutModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "LayoutView",
+      "align_content": null,
+      "align_items": null,
+      "align_self": null,
+      "border": null,
+      "bottom": null,
+      "display": null,
+      "flex": null,
+      "flex_flow": null,
+      "grid_area": null,
+      "grid_auto_columns": null,
+      "grid_auto_flow": null,
+      "grid_auto_rows": null,
+      "grid_column": null,
+      "grid_gap": null,
+      "grid_row": null,
+      "grid_template_areas": null,
+      "grid_template_columns": null,
+      "grid_template_rows": null,
+      "height": null,
+      "justify_content": null,
+      "justify_items": null,
+      "left": null,
+      "margin": null,
+      "max_height": null,
+      "max_width": null,
+      "min_height": null,
+      "min_width": null,
+      "object_fit": null,
+      "object_position": null,
+      "order": null,
+      "overflow": null,
+      "overflow_x": null,
+      "overflow_y": null,
+      "padding": null,
+      "right": null,
+      "top": null,
+      "visibility": null,
+      "width": null
+     }
+    },
+    "db5a306e5c8042e89d63144789306505": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "DescriptionStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "DescriptionStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "description_width": ""
+     }
+    },
+    "f3e0fbc3cf534e3ea70089599ccbde10": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "DescriptionStyleModel",
+     "state": {
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "DescriptionStyleModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/base",
+      "_view_module_version": "1.2.0",
+      "_view_name": "StyleView",
+      "description_width": ""
+     }
+    },
+    "f9fc8866d45d4488849542210285a0f6": {
+     "model_module": "@jupyter-widgets/controls",
+     "model_module_version": "1.5.0",
+     "model_name": "HTMLModel",
+     "state": {
+      "_dom_classes": [],
+      "_model_module": "@jupyter-widgets/controls",
+      "_model_module_version": "1.5.0",
+      "_model_name": "HTMLModel",
+      "_view_count": null,
+      "_view_module": "@jupyter-widgets/controls",
+      "_view_module_version": "1.5.0",
+      "_view_name": "HTMLView",
+      "description": "",
+      "description_tooltip": null,
+      "layout": "IPY_MODEL_57fd3787e2cd49cdbbb85d4fbfa289e4",
+      "placeholder": "​",
+      "style": "IPY_MODEL_db5a306e5c8042e89d63144789306505",
+      "value": "Loading weights: 100%"
+     }
+    }
+   }
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}