Sofiabassou1234 commited on
Commit
587ac59
·
verified ·
1 Parent(s): db03ea2

Delete datacreation.ipynb

Browse files
Files changed (1) hide show
  1. datacreation.ipynb +0 -1236
datacreation.ipynb DELETED
@@ -1,1236 +0,0 @@
1
- {
2
- "cells": [
3
- {
4
- "cell_type": "markdown",
5
- "metadata": {
6
- "id": "4ba6aba8"
7
- },
8
- "source": [
9
- "# 🤖 **Data Collection, Creation, Storage, and Processing**\n"
10
- ]
11
- },
12
- {
13
- "cell_type": "markdown",
14
- "metadata": {
15
- "id": "jpASMyIQMaAq"
16
- },
17
- "source": [
18
- "## **1.** 📦 Install required packages"
19
- ]
20
- },
21
- {
22
- "cell_type": "code",
23
- "execution_count": 6,
24
- "metadata": {
25
- "colab": {
26
- "base_uri": "https://localhost:8080/"
27
- },
28
- "id": "f48c8f8c",
29
- "outputId": "4f2a351e-5aa2-4814-c06b-520b8815dc7e"
30
- },
31
- "outputs": [
32
- {
33
- "output_type": "stream",
34
- "name": "stdout",
35
- "text": [
36
- "Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.12/dist-packages (4.13.5)\n",
37
- "Requirement already satisfied: pandas in /usr/local/lib/python3.12/dist-packages (2.2.2)\n",
38
- "Requirement already satisfied: matplotlib in /usr/local/lib/python3.12/dist-packages (3.10.0)\n",
39
- "Requirement already satisfied: seaborn in /usr/local/lib/python3.12/dist-packages (0.13.2)\n",
40
- "Requirement already satisfied: numpy in /usr/local/lib/python3.12/dist-packages (2.0.2)\n",
41
- "Requirement already satisfied: textblob in /usr/local/lib/python3.12/dist-packages (0.19.0)\n",
42
- "Requirement already satisfied: soupsieve>1.2 in /usr/local/lib/python3.12/dist-packages (from beautifulsoup4) (2.8.3)\n",
43
- "Requirement already satisfied: typing-extensions>=4.0.0 in /usr/local/lib/python3.12/dist-packages (from beautifulsoup4) (4.15.0)\n",
44
- "Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.12/dist-packages (from pandas) (2.9.0.post0)\n",
45
- "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.12/dist-packages (from pandas) (2025.2)\n",
46
- "Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.12/dist-packages (from pandas) (2025.3)\n",
47
- "Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.12/dist-packages (from matplotlib) (1.3.3)\n",
48
- "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.12/dist-packages (from matplotlib) (0.12.1)\n",
49
- "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.12/dist-packages (from matplotlib) (4.62.0)\n",
50
- "Requirement already satisfied: kiwisolver>=1.3.1 in /usr/local/lib/python3.12/dist-packages (from matplotlib) (1.5.0)\n",
51
- "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.12/dist-packages (from matplotlib) (26.0)\n",
52
- "Requirement already satisfied: pillow>=8 in /usr/local/lib/python3.12/dist-packages (from matplotlib) (11.3.0)\n",
53
- "Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.12/dist-packages (from matplotlib) (3.3.2)\n",
54
- "Requirement already satisfied: nltk>=3.9 in /usr/local/lib/python3.12/dist-packages (from textblob) (3.9.1)\n",
55
- "Requirement already satisfied: click in /usr/local/lib/python3.12/dist-packages (from nltk>=3.9->textblob) (8.3.1)\n",
56
- "Requirement already satisfied: joblib in /usr/local/lib/python3.12/dist-packages (from nltk>=3.9->textblob) (1.5.3)\n",
57
- "Requirement already satisfied: regex>=2021.8.3 in /usr/local/lib/python3.12/dist-packages (from nltk>=3.9->textblob) (2025.11.3)\n",
58
- "Requirement already satisfied: tqdm in /usr/local/lib/python3.12/dist-packages (from nltk>=3.9->textblob) (4.67.3)\n",
59
- "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.12/dist-packages (from python-dateutil>=2.8.2->pandas) (1.17.0)\n"
60
- ]
61
- }
62
- ],
63
- "source": [
64
- "!pip install beautifulsoup4 pandas matplotlib seaborn numpy textblob"
65
- ]
66
- },
67
- {
68
- "cell_type": "markdown",
69
- "metadata": {
70
- "id": "lquNYCbfL9IM"
71
- },
72
- "source": [
73
- "## **2.** ⛏ Web-scrape all book titles, prices, and ratings from books.toscrape.com"
74
- ]
75
- },
76
- {
77
- "cell_type": "markdown",
78
- "metadata": {
79
- "id": "0IWuNpxxYDJF"
80
- },
81
- "source": [
82
- "### *a. Initial setup*\n",
83
- "Define the base url of the website you will scrape as well as how and what you will scrape"
84
- ]
85
- },
86
- {
87
- "cell_type": "code",
88
- "execution_count": 4,
89
- "metadata": {
90
- "id": "91d52125"
91
- },
92
- "outputs": [],
93
- "source": [
94
- "import requests\n",
95
- "from bs4 import BeautifulSoup\n",
96
- "import pandas as pd\n",
97
- "import time\n",
98
- "\n",
99
- "base_url = \"https://books.toscrape.com/catalogue/page-{}.html\"\n",
100
- "headers = {\"User-Agent\": \"Mozilla/5.0\"}\n",
101
- "\n",
102
- "titles, prices, ratings = [], [], []"
103
- ]
104
- },
105
- {
106
- "cell_type": "markdown",
107
- "metadata": {
108
- "id": "oCdTsin2Yfp3"
109
- },
110
- "source": [
111
- "### *b. Fill titles, prices, and ratings from the web pages*"
112
- ]
113
- },
114
- {
115
- "cell_type": "code",
116
- "execution_count": 5,
117
- "metadata": {
118
- "id": "xqO5Y3dnYhxt"
119
- },
120
- "outputs": [],
121
- "source": [
122
- "# Loop through all 50 pages\n",
123
- "for page in range(1, 51):\n",
124
- " url = base_url.format(page)\n",
125
- " response = requests.get(url, headers=headers)\n",
126
- " soup = BeautifulSoup(response.content, \"html.parser\")\n",
127
- " books = soup.find_all(\"article\", class_=\"product_pod\")\n",
128
- "\n",
129
- " for book in books:\n",
130
- " titles.append(book.h3.a[\"title\"])\n",
131
- " prices.append(float(book.find(\"p\", class_=\"price_color\").text[1:]))\n",
132
- " ratings.append(book.p.get(\"class\")[1])\n",
133
- "\n",
134
- " time.sleep(0.5) # polite scraping delay"
135
- ]
136
- },
137
- {
138
- "cell_type": "markdown",
139
- "metadata": {
140
- "id": "T0TOeRC4Yrnn"
141
- },
142
- "source": [
143
- "### *c. ✋🏻🛑⛔️ Create a dataframe df_books that contains the now complete \"title\", \"price\", and \"rating\" objects*"
144
- ]
145
- },
146
- {
147
- "cell_type": "code",
148
- "execution_count": 7,
149
- "metadata": {
150
- "id": "l5FkkNhUYTHh"
151
- },
152
- "outputs": [],
153
- "source": [
154
- "import pandas as pd\n",
155
- "\n",
156
- "# Votre code existant ensuite...\n",
157
- "df_books = pd.DataFrame({\n",
158
- " \"title\": titles,\n",
159
- " \"price\": prices,\n",
160
- " \"rating\": ratings\n",
161
- "})"
162
- ]
163
- },
164
- {
165
- "cell_type": "markdown",
166
- "metadata": {
167
- "id": "duI5dv3CZYvF"
168
- },
169
- "source": [
170
- "### *d. Save web-scraped dataframe either as a CSV or Excel file*"
171
- ]
172
- },
173
- {
174
- "cell_type": "code",
175
- "execution_count": 8,
176
- "metadata": {
177
- "id": "lC1U_YHtZifh"
178
- },
179
- "outputs": [],
180
- "source": [
181
- "# 💾 Save to CSV\n",
182
- "df_books.to_csv(\"books_data.csv\", index=False)\n",
183
- "\n",
184
- "# 💾 Or save to Excel\n",
185
- "# df_books.to_excel(\"books_data.xlsx\", index=False)"
186
- ]
187
- },
188
- {
189
- "cell_type": "markdown",
190
- "metadata": {
191
- "id": "qMjRKMBQZlJi"
192
- },
193
- "source": [
194
- "### *e. ✋🏻🛑⛔️ View first fiew lines*"
195
- ]
196
- },
197
- {
198
- "cell_type": "code",
199
- "execution_count": 9,
200
- "metadata": {
201
- "colab": {
202
- "base_uri": "https://localhost:8080/",
203
- "height": 206
204
- },
205
- "id": "O_wIvTxYZqCK",
206
- "outputId": "7db18fe2-0071-4f34-f5f0-e13f8d6158e5"
207
- },
208
- "outputs": [
209
- {
210
- "output_type": "execute_result",
211
- "data": {
212
- "text/plain": [
213
- " title price rating\n",
214
- "0 A Light in the Attic 51.77 Three\n",
215
- "1 Tipping the Velvet 53.74 One\n",
216
- "2 Soumission 50.10 One\n",
217
- "3 Sharp Objects 47.82 Four\n",
218
- "4 Sapiens: A Brief History of Humankind 54.23 Five"
219
- ],
220
- "text/html": [
221
- "\n",
222
- " <div id=\"df-a53b8b01-71dc-44c2-8639-385c5b0cb6ce\" class=\"colab-df-container\">\n",
223
- " <div>\n",
224
- "<style scoped>\n",
225
- " .dataframe tbody tr th:only-of-type {\n",
226
- " vertical-align: middle;\n",
227
- " }\n",
228
- "\n",
229
- " .dataframe tbody tr th {\n",
230
- " vertical-align: top;\n",
231
- " }\n",
232
- "\n",
233
- " .dataframe thead th {\n",
234
- " text-align: right;\n",
235
- " }\n",
236
- "</style>\n",
237
- "<table border=\"1\" class=\"dataframe\">\n",
238
- " <thead>\n",
239
- " <tr style=\"text-align: right;\">\n",
240
- " <th></th>\n",
241
- " <th>title</th>\n",
242
- " <th>price</th>\n",
243
- " <th>rating</th>\n",
244
- " </tr>\n",
245
- " </thead>\n",
246
- " <tbody>\n",
247
- " <tr>\n",
248
- " <th>0</th>\n",
249
- " <td>A Light in the Attic</td>\n",
250
- " <td>51.77</td>\n",
251
- " <td>Three</td>\n",
252
- " </tr>\n",
253
- " <tr>\n",
254
- " <th>1</th>\n",
255
- " <td>Tipping the Velvet</td>\n",
256
- " <td>53.74</td>\n",
257
- " <td>One</td>\n",
258
- " </tr>\n",
259
- " <tr>\n",
260
- " <th>2</th>\n",
261
- " <td>Soumission</td>\n",
262
- " <td>50.10</td>\n",
263
- " <td>One</td>\n",
264
- " </tr>\n",
265
- " <tr>\n",
266
- " <th>3</th>\n",
267
- " <td>Sharp Objects</td>\n",
268
- " <td>47.82</td>\n",
269
- " <td>Four</td>\n",
270
- " </tr>\n",
271
- " <tr>\n",
272
- " <th>4</th>\n",
273
- " <td>Sapiens: A Brief History of Humankind</td>\n",
274
- " <td>54.23</td>\n",
275
- " <td>Five</td>\n",
276
- " </tr>\n",
277
- " </tbody>\n",
278
- "</table>\n",
279
- "</div>\n",
280
- " <div class=\"colab-df-buttons\">\n",
281
- "\n",
282
- " <div class=\"colab-df-container\">\n",
283
- " <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-a53b8b01-71dc-44c2-8639-385c5b0cb6ce')\"\n",
284
- " title=\"Convert this dataframe to an interactive table.\"\n",
285
- " style=\"display:none;\">\n",
286
- "\n",
287
- " <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
288
- " <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
289
- " </svg>\n",
290
- " </button>\n",
291
- "\n",
292
- " <style>\n",
293
- " .colab-df-container {\n",
294
- " display:flex;\n",
295
- " gap: 12px;\n",
296
- " }\n",
297
- "\n",
298
- " .colab-df-convert {\n",
299
- " background-color: #E8F0FE;\n",
300
- " border: none;\n",
301
- " border-radius: 50%;\n",
302
- " cursor: pointer;\n",
303
- " display: none;\n",
304
- " fill: #1967D2;\n",
305
- " height: 32px;\n",
306
- " padding: 0 0 0 0;\n",
307
- " width: 32px;\n",
308
- " }\n",
309
- "\n",
310
- " .colab-df-convert:hover {\n",
311
- " background-color: #E2EBFA;\n",
312
- " box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
313
- " fill: #174EA6;\n",
314
- " }\n",
315
- "\n",
316
- " .colab-df-buttons div {\n",
317
- " margin-bottom: 4px;\n",
318
- " }\n",
319
- "\n",
320
- " [theme=dark] .colab-df-convert {\n",
321
- " background-color: #3B4455;\n",
322
- " fill: #D2E3FC;\n",
323
- " }\n",
324
- "\n",
325
- " [theme=dark] .colab-df-convert:hover {\n",
326
- " background-color: #434B5C;\n",
327
- " box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
328
- " filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
329
- " fill: #FFFFFF;\n",
330
- " }\n",
331
- " </style>\n",
332
- "\n",
333
- " <script>\n",
334
- " const buttonEl =\n",
335
- " document.querySelector('#df-a53b8b01-71dc-44c2-8639-385c5b0cb6ce button.colab-df-convert');\n",
336
- " buttonEl.style.display =\n",
337
- " google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
338
- "\n",
339
- " async function convertToInteractive(key) {\n",
340
- " const element = document.querySelector('#df-a53b8b01-71dc-44c2-8639-385c5b0cb6ce');\n",
341
- " const dataTable =\n",
342
- " await google.colab.kernel.invokeFunction('convertToInteractive',\n",
343
- " [key], {});\n",
344
- " if (!dataTable) return;\n",
345
- "\n",
346
- " const docLinkHtml = 'Like what you see? Visit the ' +\n",
347
- " '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
348
- " + ' to learn more about interactive tables.';\n",
349
- " element.innerHTML = '';\n",
350
- " dataTable['output_type'] = 'display_data';\n",
351
- " await google.colab.output.renderOutput(dataTable, element);\n",
352
- " const docLink = document.createElement('div');\n",
353
- " docLink.innerHTML = docLinkHtml;\n",
354
- " element.appendChild(docLink);\n",
355
- " }\n",
356
- " </script>\n",
357
- " </div>\n",
358
- "\n",
359
- "\n",
360
- " </div>\n",
361
- " </div>\n"
362
- ],
363
- "application/vnd.google.colaboratory.intrinsic+json": {
364
- "type": "dataframe",
365
- "variable_name": "df_books",
366
- "summary": "{\n \"name\": \"df_books\",\n \"rows\": 1000,\n \"fields\": [\n {\n \"column\": \"title\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 999,\n \"samples\": [\n \"The Grownup\",\n \"Persepolis: The Story of a Childhood (Persepolis #1-2)\",\n \"Ayumi's Violin\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"price\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 14.446689669952772,\n \"min\": 10.0,\n \"max\": 59.99,\n \"num_unique_values\": 903,\n \"samples\": [\n 19.73,\n 55.65,\n 46.31\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"rating\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 5,\n \"samples\": [\n \"One\",\n \"Two\",\n \"Four\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}"
367
- }
368
- },
369
- "metadata": {},
370
- "execution_count": 9
371
- }
372
- ],
373
- "source": [
374
- "# View first few lines\n",
375
- "df_books.head()"
376
- ]
377
- },
378
- {
379
- "cell_type": "markdown",
380
- "metadata": {
381
- "id": "p-1Pr2szaqLk"
382
- },
383
- "source": [
384
- "## **3.** 🧩 Create a meaningful connection between real & synthetic datasets"
385
- ]
386
- },
387
- {
388
- "cell_type": "markdown",
389
- "metadata": {
390
- "id": "SIaJUGIpaH4V"
391
- },
392
- "source": [
393
- "### *a. Initial setup*"
394
- ]
395
- },
396
- {
397
- "cell_type": "code",
398
- "execution_count": 10,
399
- "metadata": {
400
- "id": "-gPXGcRPuV_9"
401
- },
402
- "outputs": [],
403
- "source": [
404
- "import numpy as np\n",
405
- "import random\n",
406
- "from datetime import datetime\n",
407
- "import warnings\n",
408
- "\n",
409
- "warnings.filterwarnings(\"ignore\")\n",
410
- "random.seed(2025)\n",
411
- "np.random.seed(2025)"
412
- ]
413
- },
414
- {
415
- "cell_type": "markdown",
416
- "metadata": {
417
- "id": "pY4yCoIuaQqp"
418
- },
419
- "source": [
420
- "### *b. Generate popularity scores based on rating (with some randomness) with a generate_popularity_score function*"
421
- ]
422
- },
423
- {
424
- "cell_type": "code",
425
- "execution_count": 11,
426
- "metadata": {
427
- "id": "mnd5hdAbaNjz"
428
- },
429
- "outputs": [],
430
- "source": [
431
- "def generate_popularity_score(rating):\n",
432
- " base = {\"One\": 2, \"Two\": 3, \"Three\": 3, \"Four\": 4, \"Five\": 4}.get(rating, 3)\n",
433
- " trend_factor = random.choices([-1, 0, 1], weights=[1, 3, 2])[0]\n",
434
- " return int(np.clip(base + trend_factor, 1, 5))"
435
- ]
436
- },
437
- {
438
- "cell_type": "markdown",
439
- "metadata": {
440
- "id": "n4-TaNTFgPak"
441
- },
442
- "source": [
443
- "### *c. ✋🏻🛑⛔️ Run the function to create a \"popularity_score\" column from \"rating\"*"
444
- ]
445
- },
446
- {
447
- "cell_type": "code",
448
- "execution_count": 12,
449
- "metadata": {
450
- "id": "V-G3OCUCgR07",
451
- "colab": {
452
- "base_uri": "https://localhost:8080/"
453
- },
454
- "outputId": "f7a0762f-a582-4828-99f6-7b5f73d09e64"
455
- },
456
- "outputs": [
457
- {
458
- "output_type": "stream",
459
- "name": "stdout",
460
- "text": [
461
- "popularity_score\n",
462
- "3 327\n",
463
- "4 321\n",
464
- "2 197\n",
465
- "5 117\n",
466
- "1 38\n",
467
- "Name: count, dtype: int64\n"
468
- ]
469
- }
470
- ],
471
- "source": [
472
- "# Create \"popularity_score\" column\n",
473
- "df_books['popularity_score'] = df_books['rating'].apply(generate_popularity_score)\n",
474
- "\n",
475
- "# Check the distribution\n",
476
- "print(df_books['popularity_score'].value_counts())"
477
- ]
478
- },
479
- {
480
- "cell_type": "markdown",
481
- "metadata": {
482
- "id": "HnngRNTgacYt"
483
- },
484
- "source": [
485
- "### *d. Decide on the sentiment_label based on the popularity score with a get_sentiment function*"
486
- ]
487
- },
488
- {
489
- "cell_type": "code",
490
- "execution_count": 13,
491
- "metadata": {
492
- "id": "kUtWmr8maZLZ"
493
- },
494
- "outputs": [],
495
- "source": [
496
- "def get_sentiment(popularity_score):\n",
497
- " if popularity_score <= 2:\n",
498
- " return \"negative\"\n",
499
- " elif popularity_score == 3:\n",
500
- " return \"neutral\"\n",
501
- " else:\n",
502
- " return \"positive\""
503
- ]
504
- },
505
- {
506
- "cell_type": "markdown",
507
- "metadata": {
508
- "id": "HF9F9HIzgT7Z"
509
- },
510
- "source": [
511
- "### *e. ✋🏻🛑⛔️ Run the function to create a \"sentiment_label\" column from \"popularity_score\"*"
512
- ]
513
- },
514
- {
515
- "cell_type": "code",
516
- "execution_count": 14,
517
- "metadata": {
518
- "id": "tafQj8_7gYCG",
519
- "colab": {
520
- "base_uri": "https://localhost:8080/",
521
- "height": 206
522
- },
523
- "outputId": "18b01c21-498e-44a2-9b73-821f63d3b132"
524
- },
525
- "outputs": [
526
- {
527
- "output_type": "execute_result",
528
- "data": {
529
- "text/plain": [
530
- " title price rating popularity_score \\\n",
531
- "0 A Light in the Attic 51.77 Three 3 \n",
532
- "1 Tipping the Velvet 53.74 One 2 \n",
533
- "2 Soumission 50.10 One 2 \n",
534
- "3 Sharp Objects 47.82 Four 4 \n",
535
- "4 Sapiens: A Brief History of Humankind 54.23 Five 3 \n",
536
- "\n",
537
- " sentiment_label \n",
538
- "0 neutral \n",
539
- "1 negative \n",
540
- "2 negative \n",
541
- "3 positive \n",
542
- "4 neutral "
543
- ],
544
- "text/html": [
545
- "\n",
546
- " <div id=\"df-7ddddbc1-b4b2-4a02-8eff-731f14429d31\" class=\"colab-df-container\">\n",
547
- " <div>\n",
548
- "<style scoped>\n",
549
- " .dataframe tbody tr th:only-of-type {\n",
550
- " vertical-align: middle;\n",
551
- " }\n",
552
- "\n",
553
- " .dataframe tbody tr th {\n",
554
- " vertical-align: top;\n",
555
- " }\n",
556
- "\n",
557
- " .dataframe thead th {\n",
558
- " text-align: right;\n",
559
- " }\n",
560
- "</style>\n",
561
- "<table border=\"1\" class=\"dataframe\">\n",
562
- " <thead>\n",
563
- " <tr style=\"text-align: right;\">\n",
564
- " <th></th>\n",
565
- " <th>title</th>\n",
566
- " <th>price</th>\n",
567
- " <th>rating</th>\n",
568
- " <th>popularity_score</th>\n",
569
- " <th>sentiment_label</th>\n",
570
- " </tr>\n",
571
- " </thead>\n",
572
- " <tbody>\n",
573
- " <tr>\n",
574
- " <th>0</th>\n",
575
- " <td>A Light in the Attic</td>\n",
576
- " <td>51.77</td>\n",
577
- " <td>Three</td>\n",
578
- " <td>3</td>\n",
579
- " <td>neutral</td>\n",
580
- " </tr>\n",
581
- " <tr>\n",
582
- " <th>1</th>\n",
583
- " <td>Tipping the Velvet</td>\n",
584
- " <td>53.74</td>\n",
585
- " <td>One</td>\n",
586
- " <td>2</td>\n",
587
- " <td>negative</td>\n",
588
- " </tr>\n",
589
- " <tr>\n",
590
- " <th>2</th>\n",
591
- " <td>Soumission</td>\n",
592
- " <td>50.10</td>\n",
593
- " <td>One</td>\n",
594
- " <td>2</td>\n",
595
- " <td>negative</td>\n",
596
- " </tr>\n",
597
- " <tr>\n",
598
- " <th>3</th>\n",
599
- " <td>Sharp Objects</td>\n",
600
- " <td>47.82</td>\n",
601
- " <td>Four</td>\n",
602
- " <td>4</td>\n",
603
- " <td>positive</td>\n",
604
- " </tr>\n",
605
- " <tr>\n",
606
- " <th>4</th>\n",
607
- " <td>Sapiens: A Brief History of Humankind</td>\n",
608
- " <td>54.23</td>\n",
609
- " <td>Five</td>\n",
610
- " <td>3</td>\n",
611
- " <td>neutral</td>\n",
612
- " </tr>\n",
613
- " </tbody>\n",
614
- "</table>\n",
615
- "</div>\n",
616
- " <div class=\"colab-df-buttons\">\n",
617
- "\n",
618
- " <div class=\"colab-df-container\">\n",
619
- " <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-7ddddbc1-b4b2-4a02-8eff-731f14429d31')\"\n",
620
- " title=\"Convert this dataframe to an interactive table.\"\n",
621
- " style=\"display:none;\">\n",
622
- "\n",
623
- " <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
624
- " <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
625
- " </svg>\n",
626
- " </button>\n",
627
- "\n",
628
- " <style>\n",
629
- " .colab-df-container {\n",
630
- " display:flex;\n",
631
- " gap: 12px;\n",
632
- " }\n",
633
- "\n",
634
- " .colab-df-convert {\n",
635
- " background-color: #E8F0FE;\n",
636
- " border: none;\n",
637
- " border-radius: 50%;\n",
638
- " cursor: pointer;\n",
639
- " display: none;\n",
640
- " fill: #1967D2;\n",
641
- " height: 32px;\n",
642
- " padding: 0 0 0 0;\n",
643
- " width: 32px;\n",
644
- " }\n",
645
- "\n",
646
- " .colab-df-convert:hover {\n",
647
- " background-color: #E2EBFA;\n",
648
- " box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
649
- " fill: #174EA6;\n",
650
- " }\n",
651
- "\n",
652
- " .colab-df-buttons div {\n",
653
- " margin-bottom: 4px;\n",
654
- " }\n",
655
- "\n",
656
- " [theme=dark] .colab-df-convert {\n",
657
- " background-color: #3B4455;\n",
658
- " fill: #D2E3FC;\n",
659
- " }\n",
660
- "\n",
661
- " [theme=dark] .colab-df-convert:hover {\n",
662
- " background-color: #434B5C;\n",
663
- " box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
664
- " filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
665
- " fill: #FFFFFF;\n",
666
- " }\n",
667
- " </style>\n",
668
- "\n",
669
- " <script>\n",
670
- " const buttonEl =\n",
671
- " document.querySelector('#df-7ddddbc1-b4b2-4a02-8eff-731f14429d31 button.colab-df-convert');\n",
672
- " buttonEl.style.display =\n",
673
- " google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
674
- "\n",
675
- " async function convertToInteractive(key) {\n",
676
- " const element = document.querySelector('#df-7ddddbc1-b4b2-4a02-8eff-731f14429d31');\n",
677
- " const dataTable =\n",
678
- " await google.colab.kernel.invokeFunction('convertToInteractive',\n",
679
- " [key], {});\n",
680
- " if (!dataTable) return;\n",
681
- "\n",
682
- " const docLinkHtml = 'Like what you see? Visit the ' +\n",
683
- " '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
684
- " + ' to learn more about interactive tables.';\n",
685
- " element.innerHTML = '';\n",
686
- " dataTable['output_type'] = 'display_data';\n",
687
- " await google.colab.output.renderOutput(dataTable, element);\n",
688
- " const docLink = document.createElement('div');\n",
689
- " docLink.innerHTML = docLinkHtml;\n",
690
- " element.appendChild(docLink);\n",
691
- " }\n",
692
- " </script>\n",
693
- " </div>\n",
694
- "\n",
695
- "\n",
696
- " </div>\n",
697
- " </div>\n"
698
- ],
699
- "application/vnd.google.colaboratory.intrinsic+json": {
700
- "type": "dataframe",
701
- "variable_name": "df_books",
702
- "summary": "{\n \"name\": \"df_books\",\n \"rows\": 1000,\n \"fields\": [\n {\n \"column\": \"title\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 999,\n \"samples\": [\n \"The Grownup\",\n \"Persepolis: The Story of a Childhood (Persepolis #1-2)\",\n \"Ayumi's Violin\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"price\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 14.446689669952772,\n \"min\": 10.0,\n \"max\": 59.99,\n \"num_unique_values\": 903,\n \"samples\": [\n 19.73,\n 55.65,\n 46.31\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"rating\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 5,\n \"samples\": [\n \"One\",\n \"Two\",\n \"Four\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"popularity_score\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 1,\n \"min\": 1,\n \"max\": 5,\n \"num_unique_values\": 5,\n \"samples\": [\n 2,\n 5,\n 4\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"sentiment_label\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 3,\n \"samples\": [\n \"neutral\",\n \"negative\",\n \"positive\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}"
703
- }
704
- },
705
- "metadata": {},
706
- "execution_count": 14
707
- }
708
- ],
709
- "source": [
710
- "# Create \"sentiment_label\" column\n",
711
- "df_books['sentiment_label'] = df_books['popularity_score'].apply(get_sentiment)\n",
712
- "\n",
713
- "# View the result\n",
714
- "df_books.head()"
715
- ]
716
- },
717
- {
718
- "cell_type": "markdown",
719
- "metadata": {
720
- "id": "T8AdKkmASq9a"
721
- },
722
- "source": [
723
- "## **4.** 📈 Generate synthetic book sales data of 18 months"
724
- ]
725
- },
726
- {
727
- "cell_type": "markdown",
728
- "metadata": {
729
- "id": "OhXbdGD5fH0c"
730
- },
731
- "source": [
732
- "### *a. Create a generate_sales_profit function that would generate sales patterns based on sentiment_label (with some randomness)*"
733
- ]
734
- },
735
- {
736
- "cell_type": "code",
737
- "execution_count": 15,
738
- "metadata": {
739
- "id": "qkVhYPXGbgEn"
740
- },
741
- "outputs": [],
742
- "source": [
743
- "def generate_sales_profile(sentiment):\n",
744
- " months = pd.date_range(end=datetime.today(), periods=18, freq=\"M\")\n",
745
- "\n",
746
- " if sentiment == \"positive\":\n",
747
- " base = random.randint(200, 300)\n",
748
- " trend = np.linspace(base, base + random.randint(20, 60), len(months))\n",
749
- " elif sentiment == \"negative\":\n",
750
- " base = random.randint(20, 80)\n",
751
- " trend = np.linspace(base, base - random.randint(10, 30), len(months))\n",
752
- " else: # neutral\n",
753
- " base = random.randint(80, 160)\n",
754
- " trend = np.full(len(months), base + random.randint(-10, 10))\n",
755
- "\n",
756
- " seasonality = 10 * np.sin(np.linspace(0, 3 * np.pi, len(months)))\n",
757
- " noise = np.random.normal(0, 5, len(months))\n",
758
- " monthly_sales = np.clip(trend + seasonality + noise, a_min=0, a_max=None).astype(int)\n",
759
- "\n",
760
- " return list(zip(months.strftime(\"%Y-%m\"), monthly_sales))"
761
- ]
762
- },
763
- {
764
- "cell_type": "markdown",
765
- "metadata": {
766
- "id": "L2ak1HlcgoTe"
767
- },
768
- "source": [
769
- "### *b. Run the function as part of building sales_data*"
770
- ]
771
- },
772
- {
773
- "cell_type": "code",
774
- "execution_count": 16,
775
- "metadata": {
776
- "id": "SlJ24AUafoDB"
777
- },
778
- "outputs": [],
779
- "source": [
780
- "sales_data = []\n",
781
- "for _, row in df_books.iterrows():\n",
782
- " records = generate_sales_profile(row[\"sentiment_label\"])\n",
783
- " for month, units in records:\n",
784
- " sales_data.append({\n",
785
- " \"title\": row[\"title\"],\n",
786
- " \"month\": month,\n",
787
- " \"units_sold\": units,\n",
788
- " \"sentiment_label\": row[\"sentiment_label\"]\n",
789
- " })"
790
- ]
791
- },
792
- {
793
- "cell_type": "markdown",
794
- "metadata": {
795
- "id": "4IXZKcCSgxnq"
796
- },
797
- "source": [
798
- "### *c. ✋🏻🛑⛔️ Create a df_sales DataFrame from sales_data*"
799
- ]
800
- },
801
- {
802
- "cell_type": "code",
803
- "execution_count": 17,
804
- "metadata": {
805
- "id": "wcN6gtiZg-ws",
806
- "colab": {
807
- "base_uri": "https://localhost:8080/"
808
- },
809
- "outputId": "7e35c81f-4dea-417a-c233-55b51a60db91"
810
- },
811
- "outputs": [
812
- {
813
- "output_type": "stream",
814
- "name": "stdout",
815
- "text": [
816
- "Sales dataframe created with 18000 rows.\n"
817
- ]
818
- }
819
- ],
820
- "source": [
821
- "# Create df_sales DataFrame\n",
822
- "df_sales = pd.DataFrame(sales_data)\n",
823
- "\n",
824
- "# Ensure the month column is in datetime format if needed later\n",
825
- "df_sales['month'] = pd.to_datetime(df_sales['month'])\n",
826
- "\n",
827
- "print(f\"Sales dataframe created with {len(df_sales)} rows.\")"
828
- ]
829
- },
830
- {
831
- "cell_type": "markdown",
832
- "metadata": {
833
- "id": "EhIjz9WohAmZ"
834
- },
835
- "source": [
836
- "### *d. Save df_sales as synthetic_sales_data.csv & view first few lines*"
837
- ]
838
- },
839
- {
840
- "cell_type": "code",
841
- "execution_count": 18,
842
- "metadata": {
843
- "colab": {
844
- "base_uri": "https://localhost:8080/"
845
- },
846
- "id": "MzbZvLcAhGaH",
847
- "outputId": "90dc0495-03b3-4e44-f674-894093fcfffd"
848
- },
849
- "outputs": [
850
- {
851
- "output_type": "stream",
852
- "name": "stdout",
853
- "text": [
854
- " title month units_sold sentiment_label\n",
855
- "0 A Light in the Attic 2024-09-01 100 neutral\n",
856
- "1 A Light in the Attic 2024-10-01 109 neutral\n",
857
- "2 A Light in the Attic 2024-11-01 102 neutral\n",
858
- "3 A Light in the Attic 2024-12-01 107 neutral\n",
859
- "4 A Light in the Attic 2025-01-01 108 neutral\n"
860
- ]
861
- }
862
- ],
863
- "source": [
864
- "df_sales.to_csv(\"synthetic_sales_data.csv\", index=False)\n",
865
- "\n",
866
- "print(df_sales.head())"
867
- ]
868
- },
869
- {
870
- "cell_type": "markdown",
871
- "metadata": {
872
- "id": "7g9gqBgQMtJn"
873
- },
874
- "source": [
875
- "## **5.** 🎯 Generate synthetic customer reviews"
876
- ]
877
- },
878
- {
879
- "cell_type": "markdown",
880
- "metadata": {
881
- "id": "Gi4y9M9KuDWx"
882
- },
883
- "source": [
884
- "### *a. ✋🏻🛑⛔️ Ask ChatGPT to create a list of 50 distinct generic book review texts for the sentiment labels \"positive\", \"neutral\", and \"negative\" called synthetic_reviews_by_sentiment*"
885
- ]
886
- },
887
- {
888
- "cell_type": "code",
889
- "execution_count": 24,
890
- "metadata": {
891
- "id": "b3cd2a50"
892
- },
893
- "outputs": [],
894
- "source": [
895
- "synthetic_reviews_by_sentiment = {\n",
896
- " \"positive\": [\n",
897
- " \"A compelling and heartwarming read that stayed with me long after I finished.\",\n",
898
- " \"Brilliantly written! The characters were unforgettable and the plot was engaging.\",\n",
899
- " \"One of the best books I've read this year — inspiring and emotionally rich.\",\n",
900
- " \"An absolute masterpiece of storytelling. I couldn't put it down!\",\n",
901
- " \"Stunning prose and a deep, meaningful message. Highly recommend.\",\n",
902
- " \"A refreshing and original take on the genre. Truly enjoyable.\",\n",
903
- " \"Captivating from the very first page to the last. A must-read.\",\n",
904
- " \"Beautifully crafted and thought-provoking. It touched my soul.\",\n",
905
- " \"The author's writing style is poetic and incredibly vivid.\",\n",
906
- " \"A fast-paced, exciting journey that kept me guessing until the end.\",\n",
907
- " \"Warm, witty, and wonderful. A delightful experience.\",\n",
908
- " \"Exceptional character development and a perfectly paced plot.\",\n",
909
- " \"I was completely immersed in this world. Simply brilliant.\",\n",
910
- " \"An insightful look into the human condition. Very moving.\",\n",
911
- " \"Powerful and evocative. This book deserves all the praise it gets.\",\n",
912
- " \"A charming and intelligent book that is both fun and deep.\",\n",
913
- " \"Rarely do I find a book so perfectly balanced and well-executed.\"\n",
914
- " ],\n",
915
- " \"neutral\": [\n",
916
- " \"An average book — not great, but not bad either.\",\n",
917
- " \"Some parts really stood out, others felt a bit flat.\",\n",
918
- " \"It was okay overall. A decent way to pass the time.\",\n",
919
- " \"A straightforward read with no major surprises.\",\n",
920
- " \"The premise was interesting, but the execution was just fine.\",\n",
921
- " \"I liked it, but I didn't love it. Worth a read if you're bored.\",\n",
922
- " \"A middle-of-the-road story that hits most of the expected beats.\",\n",
923
- " \"The pacing was a bit uneven, but the ending was satisfactory.\",\n",
924
- " \"It's a solid, functional story that doesn't take many risks.\",\n",
925
- " \"Some good ideas here, though they could have been explored more.\",\n",
926
- " \"Neither groundbreaking nor disappointing. Just a typical read.\",\n",
927
- " \"I found myself drifting in the middle, but the start was strong.\",\n",
928
- " \"Decent prose, but the plot felt a little predictable at times.\",\n",
929
- " \"An easy read for a rainy afternoon, but forgettable.\",\n",
930
- " \"The characters were relatable enough, if slightly generic.\",\n",
931
- " \"It fulfills its promise without exceeding any expectations.\",\n",
932
- " \"Standard fare with some promise, though not quite polished.\"\n",
933
- " ],\n",
934
- " \"negative\": [\n",
935
- " \"I struggled to get through this one — it just didn’t grab me.\",\n",
936
- " \"The plot was confusing and the characters felt underdeveloped.\",\n",
937
- " \"Disappointing. I had high hopes, but they weren't met.\",\n",
938
- " \"The writing felt forced and the dialogue was quite clichéd.\",\n",
939
- " \"I found it difficult to care about what happened to the characters.\",\n",
940
- " \"The story dragged on far too long without much progression.\",\n",
941
- " \"Lackluster and repetitive. It failed to hold my interest.\",\n",
942
- " \"A frustrating read with too many plot holes to count.\",\n",
943
- " \"The tone was inconsistent and the ending felt very rushed.\",\n",
944
- " \"I didn't connect with the style at all. It felt quite dry.\",\n",
945
- " \"Had potential that went unrealized. Quite a letdown.\",\n",
946
- " \"It simply lacked that emotional punch I was looking for.\",\n",
947
- " \"The themes were solid, but they were not well explored.\",\n",
948
- " \"I expected much more depth from a book with this premise.\",\n",
949
- " \"Hard to follow and ultimately not very rewarding.\",\n",
950
- " \"Serviceable but not something I'd ever go out of my way for.\"\n",
951
- " ]\n",
952
- "}"
953
- ]
954
- },
955
- {
956
- "cell_type": "markdown",
957
- "metadata": {
958
- "id": "fQhfVaDmuULT"
959
- },
960
- "source": [
961
- "### *b. Generate 10 reviews per book using random sampling from the corresponding 50*"
962
- ]
963
- },
964
- {
965
- "cell_type": "code",
966
- "execution_count": 25,
967
- "metadata": {
968
- "id": "l2SRc3PjuTGM"
969
- },
970
- "outputs": [],
971
- "source": [
972
- "review_rows = []\n",
973
- "for _, row in df_books.iterrows():\n",
974
- " title = row['title']\n",
975
- " sentiment_label = row['sentiment_label']\n",
976
- " review_pool = synthetic_reviews_by_sentiment[sentiment_label]\n",
977
- " sampled_reviews = random.sample(review_pool, 10)\n",
978
- " for review_text in sampled_reviews:\n",
979
- " review_rows.append({\n",
980
- " \"title\": title,\n",
981
- " \"sentiment_label\": sentiment_label,\n",
982
- " \"review_text\": review_text,\n",
983
- " \"rating\": row['rating'],\n",
984
- " \"popularity_score\": row['popularity_score']\n",
985
- " })"
986
- ]
987
- },
988
- {
989
- "cell_type": "markdown",
990
- "metadata": {
991
- "id": "bmJMXF-Bukdm"
992
- },
993
- "source": [
994
- "### *c. Create the final dataframe df_reviews & save it as synthetic_book_reviews.csv*"
995
- ]
996
- },
997
- {
998
- "cell_type": "code",
999
- "execution_count": 26,
1000
- "metadata": {
1001
- "id": "ZUKUqZsuumsp"
1002
- },
1003
- "outputs": [],
1004
- "source": [
1005
- "df_reviews = pd.DataFrame(review_rows)\n",
1006
- "df_reviews.to_csv(\"synthetic_book_reviews.csv\", index=False)"
1007
- ]
1008
- },
1009
- {
1010
- "cell_type": "markdown",
1011
- "source": [
1012
- "### *c. inputs for R*"
1013
- ],
1014
- "metadata": {
1015
- "id": "_602pYUS3gY5"
1016
- }
1017
- },
1018
- {
1019
- "cell_type": "code",
1020
- "execution_count": 27,
1021
- "metadata": {
1022
- "colab": {
1023
- "base_uri": "https://localhost:8080/"
1024
- },
1025
- "id": "3946e521",
1026
- "outputId": "876c6b3d-301e-4506-d432-12c4ad8ac87a"
1027
- },
1028
- "outputs": [
1029
- {
1030
- "output_type": "stream",
1031
- "name": "stdout",
1032
- "text": [
1033
- "✅ Wrote synthetic_title_level_features.csv\n",
1034
- "✅ Wrote synthetic_monthly_revenue_series.csv\n"
1035
- ]
1036
- }
1037
- ],
1038
- "source": [
1039
- "import numpy as np\n",
1040
- "\n",
1041
- "def _safe_num(s):\n",
1042
- " return pd.to_numeric(\n",
1043
- " pd.Series(s).astype(str).str.replace(r\"[^0-9.]\", \"\", regex=True),\n",
1044
- " errors=\"coerce\"\n",
1045
- " )\n",
1046
- "\n",
1047
- "# --- Clean book metadata (price/rating) ---\n",
1048
- "df_books_r = df_books.copy()\n",
1049
- "if \"price\" in df_books_r.columns:\n",
1050
- " df_books_r[\"price\"] = _safe_num(df_books_r[\"price\"])\n",
1051
- "if \"rating\" in df_books_r.columns:\n",
1052
- " df_books_r[\"rating\"] = _safe_num(df_books_r[\"rating\"])\n",
1053
- "\n",
1054
- "df_books_r[\"title\"] = df_books_r[\"title\"].astype(str).str.strip()\n",
1055
- "\n",
1056
- "# --- Clean sales ---\n",
1057
- "df_sales_r = df_sales.copy()\n",
1058
- "df_sales_r[\"title\"] = df_sales_r[\"title\"].astype(str).str.strip()\n",
1059
- "df_sales_r[\"month\"] = pd.to_datetime(df_sales_r[\"month\"], errors=\"coerce\")\n",
1060
- "df_sales_r[\"units_sold\"] = _safe_num(df_sales_r[\"units_sold\"])\n",
1061
- "\n",
1062
- "# --- Clean reviews ---\n",
1063
- "df_reviews_r = df_reviews.copy()\n",
1064
- "df_reviews_r[\"title\"] = df_reviews_r[\"title\"].astype(str).str.strip()\n",
1065
- "df_reviews_r[\"sentiment_label\"] = df_reviews_r[\"sentiment_label\"].astype(str).str.lower().str.strip()\n",
1066
- "if \"rating\" in df_reviews_r.columns:\n",
1067
- " df_reviews_r[\"rating\"] = _safe_num(df_reviews_r[\"rating\"])\n",
1068
- "if \"popularity_score\" in df_reviews_r.columns:\n",
1069
- " df_reviews_r[\"popularity_score\"] = _safe_num(df_reviews_r[\"popularity_score\"])\n",
1070
- "\n",
1071
- "# --- Sentiment shares per title (from reviews) ---\n",
1072
- "sent_counts = (\n",
1073
- " df_reviews_r.groupby([\"title\", \"sentiment_label\"])\n",
1074
- " .size()\n",
1075
- " .unstack(fill_value=0)\n",
1076
- ")\n",
1077
- "for lab in [\"positive\", \"neutral\", \"negative\"]:\n",
1078
- " if lab not in sent_counts.columns:\n",
1079
- " sent_counts[lab] = 0\n",
1080
- "\n",
1081
- "sent_counts[\"total_reviews\"] = sent_counts[[\"positive\", \"neutral\", \"negative\"]].sum(axis=1)\n",
1082
- "den = sent_counts[\"total_reviews\"].replace(0, np.nan)\n",
1083
- "sent_counts[\"share_positive\"] = sent_counts[\"positive\"] / den\n",
1084
- "sent_counts[\"share_neutral\"] = sent_counts[\"neutral\"] / den\n",
1085
- "sent_counts[\"share_negative\"] = sent_counts[\"negative\"] / den\n",
1086
- "sent_counts = sent_counts.reset_index()\n",
1087
- "\n",
1088
- "# --- Sales aggregation per title ---\n",
1089
- "sales_by_title = (\n",
1090
- " df_sales_r.dropna(subset=[\"title\"])\n",
1091
- " .groupby(\"title\", as_index=False)\n",
1092
- " .agg(\n",
1093
- " months_observed=(\"month\", \"nunique\"),\n",
1094
- " avg_units_sold=(\"units_sold\", \"mean\"),\n",
1095
- " total_units_sold=(\"units_sold\", \"sum\"),\n",
1096
- " )\n",
1097
- ")\n",
1098
- "\n",
1099
- "# --- Title-level features (join sales + books + sentiment) ---\n",
1100
- "df_title = (\n",
1101
- " sales_by_title\n",
1102
- " .merge(df_books_r[[\"title\", \"price\", \"rating\"]], on=\"title\", how=\"left\")\n",
1103
- " .merge(sent_counts[[\"title\", \"share_positive\", \"share_neutral\", \"share_negative\", \"total_reviews\"]],\n",
1104
- " on=\"title\", how=\"left\")\n",
1105
- ")\n",
1106
- "\n",
1107
- "df_title[\"avg_revenue\"] = df_title[\"avg_units_sold\"] * df_title[\"price\"]\n",
1108
- "df_title[\"total_revenue\"] = df_title[\"total_units_sold\"] * df_title[\"price\"]\n",
1109
- "\n",
1110
- "df_title.to_csv(\"synthetic_title_level_features.csv\", index=False)\n",
1111
- "print(\"✅ Wrote synthetic_title_level_features.csv\")\n",
1112
- "\n",
1113
- "# --- Monthly revenue series (proxy: units_sold * price) ---\n",
1114
- "monthly_rev = (\n",
1115
- " df_sales_r.merge(df_books_r[[\"title\", \"price\"]], on=\"title\", how=\"left\")\n",
1116
- ")\n",
1117
- "monthly_rev[\"revenue\"] = monthly_rev[\"units_sold\"] * monthly_rev[\"price\"]\n",
1118
- "\n",
1119
- "df_monthly = (\n",
1120
- " monthly_rev.dropna(subset=[\"month\"])\n",
1121
- " .groupby(\"month\", as_index=False)[\"revenue\"]\n",
1122
- " .sum()\n",
1123
- " .rename(columns={\"revenue\": \"total_revenue\"})\n",
1124
- " .sort_values(\"month\")\n",
1125
- ")\n",
1126
- "# if revenue is all NA (e.g., missing price), fallback to units_sold as a teaching proxy\n",
1127
- "if df_monthly[\"total_revenue\"].notna().sum() == 0:\n",
1128
- " df_monthly = (\n",
1129
- " df_sales_r.dropna(subset=[\"month\"])\n",
1130
- " .groupby(\"month\", as_index=False)[\"units_sold\"]\n",
1131
- " .sum()\n",
1132
- " .rename(columns={\"units_sold\": \"total_revenue\"})\n",
1133
- " .sort_values(\"month\")\n",
1134
- " )\n",
1135
- "\n",
1136
- "df_monthly[\"month\"] = pd.to_datetime(df_monthly[\"month\"], errors=\"coerce\").dt.strftime(\"%Y-%m-%d\")\n",
1137
- "df_monthly.to_csv(\"synthetic_monthly_revenue_series.csv\", index=False)\n",
1138
- "print(\"✅ Wrote synthetic_monthly_revenue_series.csv\")\n"
1139
- ]
1140
- },
1141
- {
1142
- "cell_type": "markdown",
1143
- "metadata": {
1144
- "id": "RYvGyVfXuo54"
1145
- },
1146
- "source": [
1147
- "### *d. ✋🏻🛑⛔️ View the first few lines*"
1148
- ]
1149
- },
1150
- {
1151
- "cell_type": "code",
1152
- "execution_count": 28,
1153
- "metadata": {
1154
- "colab": {
1155
- "base_uri": "https://localhost:8080/"
1156
- },
1157
- "id": "xfE8NMqOurKo",
1158
- "outputId": "3eccca7b-e975-4860-8b9f-09375f975c06"
1159
- },
1160
- "outputs": [
1161
- {
1162
- "output_type": "stream",
1163
- "name": "stdout",
1164
- "text": [
1165
- " title sentiment_label \\\n",
1166
- "0 A Light in the Attic neutral \n",
1167
- "1 A Light in the Attic neutral \n",
1168
- "2 A Light in the Attic neutral \n",
1169
- "3 A Light in the Attic neutral \n",
1170
- "4 A Light in the Attic neutral \n",
1171
- "\n",
1172
- " review_text rating popularity_score \n",
1173
- "0 I found myself drifting in the middle, but the... Three 3 \n",
1174
- "1 The premise was interesting, but the execution... Three 3 \n",
1175
- "2 Standard fare with some promise, though not qu... Three 3 \n",
1176
- "3 A middle-of-the-road story that hits most of t... Three 3 \n",
1177
- "4 Some good ideas here, though they could have b... Three 3 \n",
1178
- "\n",
1179
- "Sentiment distribution in reviews:\n",
1180
- "sentiment_label\n",
1181
- "positive 4380\n",
1182
- "neutral 3270\n",
1183
- "negative 2350\n",
1184
- "Name: count, dtype: int64\n"
1185
- ]
1186
- }
1187
- ],
1188
- "source": [
1189
- "# View the first few lines of the reviews dataframe\n",
1190
- "print(df_reviews.head())\n",
1191
- "\n",
1192
- "# Optional: Check a summary of sentiments\n",
1193
- "print(\"\\nSentiment distribution in reviews:\")\n",
1194
- "print(df_reviews['sentiment_label'].value_counts())"
1195
- ]
1196
- }
1197
- ],
1198
- "metadata": {
1199
- "colab": {
1200
- "collapsed_sections": [
1201
- "jpASMyIQMaAq",
1202
- "lquNYCbfL9IM",
1203
- "0IWuNpxxYDJF",
1204
- "oCdTsin2Yfp3",
1205
- "T0TOeRC4Yrnn",
1206
- "duI5dv3CZYvF",
1207
- "qMjRKMBQZlJi",
1208
- "p-1Pr2szaqLk",
1209
- "SIaJUGIpaH4V",
1210
- "pY4yCoIuaQqp",
1211
- "n4-TaNTFgPak",
1212
- "HnngRNTgacYt",
1213
- "HF9F9HIzgT7Z",
1214
- "T8AdKkmASq9a",
1215
- "OhXbdGD5fH0c",
1216
- "L2ak1HlcgoTe",
1217
- "4IXZKcCSgxnq",
1218
- "EhIjz9WohAmZ",
1219
- "Gi4y9M9KuDWx",
1220
- "fQhfVaDmuULT",
1221
- "bmJMXF-Bukdm",
1222
- "RYvGyVfXuo54"
1223
- ],
1224
- "provenance": []
1225
- },
1226
- "kernelspec": {
1227
- "display_name": "Python 3",
1228
- "name": "python3"
1229
- },
1230
- "language_info": {
1231
- "name": "python"
1232
- }
1233
- },
1234
- "nbformat": 4,
1235
- "nbformat_minor": 0
1236
- }