Upload 2 files

evaluation_results/classification_report.csv

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7726c5b7899421298a3732569702f85b7584dd4e9b89229a46b8433c556ee026
+size 400

evaluation_results/evaluation_results.txt

@@ -0,0 +1,144 @@
+✅ Loaded 1953 rows from 7 CSV files.
+Columns detected in CSVs: ['tweet_id', 'datetime', 'username', 'user_display_name', 'user_followers', 'user_following', 'user_verified', 'content', 'user_location', 'user_description', 'hashtags', 'mentions', 'phone_numbers', 'tweet_url', 'retweet_count', 'like_count', 'reply_count', 'kar_score', 'drug_score', 'crime_score', 'sentiment', 'sentiment_compound', 'is_drug_related', 'is_crime_related', 'has_contact_info', 'risk_level', 'content_hash', 'date_parsed']
+
+=== General Stats ===
+Columns: ['tweet_id', 'datetime', 'username', 'user_display_name', 'user_followers', 'user_following', 'user_verified', 'content', 'user_location', 'user_description', 'hashtags', 'mentions', 'phone_numbers', 'tweet_url', 'retweet_count', 'like_count', 'reply_count', 'kar_score', 'drug_score', 'crime_score', 'sentiment', 'sentiment_compound', 'is_drug_related', 'is_crime_related', 'has_contact_info', 'risk_level', 'content_hash', 'date_parsed']
+Total rows: 1953
+Missing values per column:
+tweet_id                 0
+datetime                 0
+username                 0
+user_display_name        0
+user_followers           0
+user_following           0
+user_verified            0
+content                  0
+user_location          362
+user_description      1953
+hashtags               978
+mentions              1349
+phone_numbers         1939
+tweet_url                0
+retweet_count            0
+like_count               0
+reply_count              0
+kar_score                0
+drug_score               0
+crime_score              0
+sentiment                0
+sentiment_compound       0
+is_drug_related          0
+is_crime_related         0
+has_contact_info         0
+risk_level               0
+content_hash             0
+date_parsed              0
+dtype: int64
+Duplicate rows: 1148
+
+Sample rows with missing values:
+              tweet_id             datetime         username  ... risk_level                      content_hash          date_parsed
+0  1959959601048420576  25-08-2025 18:12:37     NewsMeter_In  ...       HIGH  8abf18065977e4493f16f4b165624ef2  2025-08-25 18:12:37
+1  1966399847793345021  12-09-2025 12:43:52       idencies05  ...     MEDIUM  d555008fafd825e05aca84ccffb13a77  2025-09-12 12:43:52
+2  1969293025831530539  20-09-2025 12:20:19       idencies05  ...   CRITICAL  9884c305ea49804875088cd5b72c1781  2025-09-20 12:20:19
+3  1963241947126018435  03-09-2025 19:35:30  Prathikthethith  ...   CRITICAL  f2de5dc092522d4e1af0e49442e66937  2025-09-03 19:35:30
+4  1969293025831530539  20-09-2025 12:20:19       idencies05  ...   CRITICAL  9884c305ea49804875088cd5b72c1781  2025-09-20 12:20:19
+
+[5 rows x 28 columns]
+
+Sample duplicate rows:
+              tweet_id             datetime         username  ... risk_level                      content_hash          date_parsed
+0  1959959601048420576  25-08-2025 18:12:37     NewsMeter_In  ...       HIGH  8abf18065977e4493f16f4b165624ef2  2025-08-25 18:12:37
+1  1966399847793345021  12-09-2025 12:43:52       idencies05  ...     MEDIUM  d555008fafd825e05aca84ccffb13a77  2025-09-12 12:43:52
+2  1969293025831530539  20-09-2025 12:20:19       idencies05  ...   CRITICAL  9884c305ea49804875088cd5b72c1781  2025-09-20 12:20:19
+3  1963241947126018435  03-09-2025 19:35:30  Prathikthethith  ...   CRITICAL  f2de5dc092522d4e1af0e49442e66937  2025-09-03 19:35:30
+5  1978329389067641222  15-10-2025 10:47:36  XpressBengaluru  ...   CRITICAL  eef36dce5060c43923727565aa695b93  2025-10-15 10:47:36
+
+[5 rows x 28 columns]
+
+=== is_drug_related Distribution ===
+is_drug_related
+True     1473
+False     480
+Name: count, dtype: int64
+Proportion:
+is_drug_related
+True     0.7542
+False    0.2458
+Name: proportion, dtype: float64
+
+=== is_crime_related Distribution ===
+is_crime_related
+True     1576
+False     377
+Name: count, dtype: int64
+Proportion:
+is_crime_related
+True     0.807
+False    0.193
+Name: proportion, dtype: float64
+
+=== risk_level Distribution ===
+risk_level
+MEDIUM      1523
+LOW          290
+HIGH         127
+CRITICAL      13
+Name: count, dtype: int64
+Proportion:
+risk_level
+MEDIUM      0.7798
+LOW         0.1485
+HIGH        0.0650
+CRITICAL    0.0067
+Name: proportion, dtype: float64
+
+=== Date Range ===
+Earliest: 2025-03-14 16:42:38
+Latest: 2025-10-17 00:36:41
+
+=== Daily Counts of Posts ===
+date
+2025-03-14      2
+2025-07-19    112
+2025-07-20     35
+2025-07-21     23
+2025-07-22     27
+             ... 
+2025-10-13     43
+2025-10-14     27
+2025-10-15     20
+2025-10-16     10
+2025-10-17      4
+Length: 91, dtype: int64
+
+=== User Analysis ===
+Total unique users: 554
+Top 10 users by post count:
+username
+Newskarnataka      118
+grok                57
+KannadaRepublic     30
+XpressBengaluru     25
+path2shah           23
+bangalore_22532     21
+ndtv                20
+ians_india          18
+UvEnglish           17
+wegro_app           17
+Name: count, dtype: int64
+
+=== Scraper Evaluation Metrics ===
+Completeness (all columns filled): 88.38%
+Duplicate rows rate: 58.78%
+is_drug_related relevance rate: 75.42%
+is_crime_related relevance rate: 80.7%
+Time coverage ratio (active days / total days): 41.94%
+
+=== Classification Metrics (is_drug_related vs is_crime_related) ===
+Accuracy: 0.6994
+Precision: 0.8357
+Recall: 0.7811
+F1-score: 0.8075
+
+Classification report saved as 'classification_report.csv'