Hugging Face's logo

Spaces:
ESCP
/
GroupG-8-SE21
Sleeping

App Files Community
penelope1234 commited on
Commit
f89b776
·
verified ·
1 Parent(s): 413cccd

Delete datacreation.ipynb

Browse files
Files changed (1) hide show
  1. datacreation.ipynb +0 -1426
datacreation.ipynb DELETED
@@ -1,1426 +0,0 @@
1
- {
2
- "cells": [
3
- {
4
- "cell_type": "markdown",
5
- "metadata": {
6
- "id": "4ba6aba8"
7
- },
8
- "source": [
9
- "# 🤖 **Data Collection, Creation, Storage, and Processing**\n"
10
- ]
11
- },
12
- {
13
- "cell_type": "markdown",
14
- "metadata": {
15
- "id": "jpASMyIQMaAq"
16
- },
17
- "source": [
18
- "## **1.** 📦 Install required packages"
19
- ]
20
- },
21
- {
22
- "cell_type": "code",
23
- "execution_count": null,
24
- "metadata": {
25
- "colab": {
26
- "base_uri": "https://localhost:8080/"
27
- },
28
- "id": "f48c8f8c",
29
- "outputId": "eec32c12-8d46-423f-ed13-16e296fc1f65"
30
- },
31
- "outputs": [
32
- {
33
- "output_type": "stream",
34
- "name": "stdout",
35
- "text": [
36
- "Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.12/dist-packages (4.13.5)\n",
37
- "Requirement already satisfied: pandas in /usr/local/lib/python3.12/dist-packages (2.2.2)\n",
38
- "Requirement already satisfied: matplotlib in /usr/local/lib/python3.12/dist-packages (3.10.0)\n",
39
- "Requirement already satisfied: seaborn in /usr/local/lib/python3.12/dist-packages (0.13.2)\n",
40
- "Requirement already satisfied: numpy in /usr/local/lib/python3.12/dist-packages (2.0.2)\n",
41
- "Requirement already satisfied: textblob in /usr/local/lib/python3.12/dist-packages (0.19.0)\n",
42
- "Requirement already satisfied: soupsieve>1.2 in /usr/local/lib/python3.12/dist-packages (from beautifulsoup4) (2.8.3)\n",
43
- "Requirement already satisfied: typing-extensions>=4.0.0 in /usr/local/lib/python3.12/dist-packages (from beautifulsoup4) (4.15.0)\n",
44
- "Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.12/dist-packages (from pandas) (2.9.0.post0)\n",
45
- "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.12/dist-packages (from pandas) (2025.2)\n",
46
- "Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.12/dist-packages (from pandas) (2025.3)\n",
47
- "Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.12/dist-packages (from matplotlib) (1.3.3)\n",
48
- "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.12/dist-packages (from matplotlib) (0.12.1)\n",
49
- "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.12/dist-packages (from matplotlib) (4.61.1)\n",
50
- "Requirement already satisfied: kiwisolver>=1.3.1 in /usr/local/lib/python3.12/dist-packages (from matplotlib) (1.4.9)\n",
51
- "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.12/dist-packages (from matplotlib) (26.0)\n",
52
- "Requirement already satisfied: pillow>=8 in /usr/local/lib/python3.12/dist-packages (from matplotlib) (11.3.0)\n",
53
- "Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.12/dist-packages (from matplotlib) (3.3.2)\n",
54
- "Requirement already satisfied: nltk>=3.9 in /usr/local/lib/python3.12/dist-packages (from textblob) (3.9.1)\n",
55
- "Requirement already satisfied: click in /usr/local/lib/python3.12/dist-packages (from nltk>=3.9->textblob) (8.3.1)\n",
56
- "Requirement already satisfied: joblib in /usr/local/lib/python3.12/dist-packages (from nltk>=3.9->textblob) (1.5.3)\n",
57
- "Requirement already satisfied: regex>=2021.8.3 in /usr/local/lib/python3.12/dist-packages (from nltk>=3.9->textblob) (2025.11.3)\n",
58
- "Requirement already satisfied: tqdm in /usr/local/lib/python3.12/dist-packages (from nltk>=3.9->textblob) (4.67.3)\n",
59
- "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.12/dist-packages (from python-dateutil>=2.8.2->pandas) (1.17.0)\n"
60
- ]
61
- }
62
- ],
63
- "source": [
64
- "!pip install beautifulsoup4 pandas matplotlib seaborn numpy textblob"
65
- ]
66
- },
67
- {
68
- "cell_type": "markdown",
69
- "metadata": {
70
- "id": "lquNYCbfL9IM"
71
- },
72
- "source": [
73
- "## **2.** ⛏ Web-scrape all book titles, prices, and ratings from books.toscrape.com"
74
- ]
75
- },
76
- {
77
- "cell_type": "markdown",
78
- "metadata": {
79
- "id": "0IWuNpxxYDJF"
80
- },
81
- "source": [
82
- "### *a. Initial setup*\n",
83
- "Define the base url of the website you will scrape as well as how and what you will scrape"
84
- ]
85
- },
86
- {
87
- "cell_type": "code",
88
- "execution_count": null,
89
- "metadata": {
90
- "id": "91d52125"
91
- },
92
- "outputs": [],
93
- "source": [
94
- "import requests\n",
95
- "from bs4 import BeautifulSoup\n",
96
- "import pandas as pd\n",
97
- "import time\n",
98
- "\n",
99
- "base_url = \"https://books.toscrape.com/catalogue/page-{}.html\"\n",
100
- "headers = {\"User-Agent\": \"Mozilla/5.0\"}\n",
101
- "\n",
102
- "titles, prices, ratings = [], [], []"
103
- ]
104
- },
105
- {
106
- "cell_type": "markdown",
107
- "metadata": {
108
- "id": "oCdTsin2Yfp3"
109
- },
110
- "source": [
111
- "### *b. Fill titles, prices, and ratings from the web pages*"
112
- ]
113
- },
114
- {
115
- "cell_type": "code",
116
- "execution_count": null,
117
- "metadata": {
118
- "id": "xqO5Y3dnYhxt"
119
- },
120
- "outputs": [],
121
- "source": [
122
- "# Loop through all 50 pages\n",
123
- "for page in range(1, 51):\n",
124
- " url = base_url.format(page)\n",
125
- " response = requests.get(url, headers=headers)\n",
126
- " soup = BeautifulSoup(response.content, \"html.parser\")\n",
127
- " books = soup.find_all(\"article\", class_=\"product_pod\")\n",
128
- "\n",
129
- " for book in books:\n",
130
- " titles.append(book.h3.a[\"title\"])\n",
131
- " prices.append(float(book.find(\"p\", class_=\"price_color\").text[1:]))\n",
132
- " ratings.append(book.p.get(\"class\")[1])\n",
133
- "\n",
134
- " time.sleep(0.5) # polite scraping delay"
135
- ]
136
- },
137
- {
138
- "cell_type": "markdown",
139
- "metadata": {
140
- "id": "T0TOeRC4Yrnn"
141
- },
142
- "source": [
143
- "### *c. ✋🏻🛑⛔️ Create a dataframe df_books that contains the now complete \"title\", \"price\", and \"rating\" objects*"
144
- ]
145
- },
146
- {
147
- "cell_type": "code",
148
- "execution_count": null,
149
- "metadata": {
150
- "id": "l5FkkNhUYTHh"
151
- },
152
- "outputs": [],
153
- "source": [
154
- "# Create the dataframe\n",
155
- "df_books = pd.DataFrame({\n",
156
- " \"title\": titles,\n",
157
- " \"price\": prices,\n",
158
- " \"rating\": ratings\n",
159
- "})\n"
160
- ]
161
- },
162
- {
163
- "cell_type": "markdown",
164
- "metadata": {
165
- "id": "duI5dv3CZYvF"
166
- },
167
- "source": [
168
- "### *d. Save web-scraped dataframe either as a CSV or Excel file*"
169
- ]
170
- },
171
- {
172
- "cell_type": "code",
173
- "execution_count": null,
174
- "metadata": {
175
- "id": "lC1U_YHtZifh"
176
- },
177
- "outputs": [],
178
- "source": [
179
- "# 💾 Save to CSV\n",
180
- "df_books.to_csv(\"books_data.csv\", index=False)\n",
181
- "\n",
182
- "# 💾 Or save to Excel\n",
183
- "# df_books.to_excel(\"books_data.xlsx\", index=False)"
184
- ]
185
- },
186
- {
187
- "cell_type": "markdown",
188
- "metadata": {
189
- "id": "qMjRKMBQZlJi"
190
- },
191
- "source": [
192
- "### *e. ✋🏻🛑⛔️ View first fiew lines*"
193
- ]
194
- },
195
- {
196
- "cell_type": "code",
197
- "execution_count": null,
198
- "metadata": {
199
- "colab": {
200
- "base_uri": "https://localhost:8080/",
201
- "height": 204
202
- },
203
- "id": "O_wIvTxYZqCK",
204
- "outputId": "0a506ff9-0e46-417e-db52-14937356d9c3"
205
- },
206
- "outputs": [
207
- {
208
- "output_type": "execute_result",
209
- "data": {
210
- "text/plain": [
211
- " title price rating\n",
212
- "0 A Light in the Attic 51.77 Three\n",
213
- "1 Tipping the Velvet 53.74 One\n",
214
- "2 Soumission 50.10 One\n",
215
- "3 Sharp Objects 47.82 Four\n",
216
- "4 Sapiens: A Brief History of Humankind 54.23 Five"
217
- ],
218
- "text/html": [
219
- "\n",
220
- " <div id=\"df-b1647ad0-8c97-4561-b40f-6130d118573f\" class=\"colab-df-container\">\n",
221
- " <div>\n",
222
- "<style scoped>\n",
223
- " .dataframe tbody tr th:only-of-type {\n",
224
- " vertical-align: middle;\n",
225
- " }\n",
226
- "\n",
227
- " .dataframe tbody tr th {\n",
228
- " vertical-align: top;\n",
229
- " }\n",
230
- "\n",
231
- " .dataframe thead th {\n",
232
- " text-align: right;\n",
233
- " }\n",
234
- "</style>\n",
235
- "<table border=\"1\" class=\"dataframe\">\n",
236
- " <thead>\n",
237
- " <tr style=\"text-align: right;\">\n",
238
- " <th></th>\n",
239
- " <th>title</th>\n",
240
- " <th>price</th>\n",
241
- " <th>rating</th>\n",
242
- " </tr>\n",
243
- " </thead>\n",
244
- " <tbody>\n",
245
- " <tr>\n",
246
- " <th>0</th>\n",
247
- " <td>A Light in the Attic</td>\n",
248
- " <td>51.77</td>\n",
249
- " <td>Three</td>\n",
250
- " </tr>\n",
251
- " <tr>\n",
252
- " <th>1</th>\n",
253
- " <td>Tipping the Velvet</td>\n",
254
- " <td>53.74</td>\n",
255
- " <td>One</td>\n",
256
- " </tr>\n",
257
- " <tr>\n",
258
- " <th>2</th>\n",
259
- " <td>Soumission</td>\n",
260
- " <td>50.10</td>\n",
261
- " <td>One</td>\n",
262
- " </tr>\n",
263
- " <tr>\n",
264
- " <th>3</th>\n",
265
- " <td>Sharp Objects</td>\n",
266
- " <td>47.82</td>\n",
267
- " <td>Four</td>\n",
268
- " </tr>\n",
269
- " <tr>\n",
270
- " <th>4</th>\n",
271
- " <td>Sapiens: A Brief History of Humankind</td>\n",
272
- " <td>54.23</td>\n",
273
- " <td>Five</td>\n",
274
- " </tr>\n",
275
- " </tbody>\n",
276
- "</table>\n",
277
- "</div>\n",
278
- " <div class=\"colab-df-buttons\">\n",
279
- "\n",
280
- " <div class=\"colab-df-container\">\n",
281
- " <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-b1647ad0-8c97-4561-b40f-6130d118573f')\"\n",
282
- " title=\"Convert this dataframe to an interactive table.\"\n",
283
- " style=\"display:none;\">\n",
284
- "\n",
285
- " <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
286
- " <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
287
- " </svg>\n",
288
- " </button>\n",
289
- "\n",
290
- " <style>\n",
291
- " .colab-df-container {\n",
292
- " display:flex;\n",
293
- " gap: 12px;\n",
294
- " }\n",
295
- "\n",
296
- " .colab-df-convert {\n",
297
- " background-color: #E8F0FE;\n",
298
- " border: none;\n",
299
- " border-radius: 50%;\n",
300
- " cursor: pointer;\n",
301
- " display: none;\n",
302
- " fill: #1967D2;\n",
303
- " height: 32px;\n",
304
- " padding: 0 0 0 0;\n",
305
- " width: 32px;\n",
306
- " }\n",
307
- "\n",
308
- " .colab-df-convert:hover {\n",
309
- " background-color: #E2EBFA;\n",
310
- " box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
311
- " fill: #174EA6;\n",
312
- " }\n",
313
- "\n",
314
- " .colab-df-buttons div {\n",
315
- " margin-bottom: 4px;\n",
316
- " }\n",
317
- "\n",
318
- " [theme=dark] .colab-df-convert {\n",
319
- " background-color: #3B4455;\n",
320
- " fill: #D2E3FC;\n",
321
- " }\n",
322
- "\n",
323
- " [theme=dark] .colab-df-convert:hover {\n",
324
- " background-color: #434B5C;\n",
325
- " box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
326
- " filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
327
- " fill: #FFFFFF;\n",
328
- " }\n",
329
- " </style>\n",
330
- "\n",
331
- " <script>\n",
332
- " const buttonEl =\n",
333
- " document.querySelector('#df-b1647ad0-8c97-4561-b40f-6130d118573f button.colab-df-convert');\n",
334
- " buttonEl.style.display =\n",
335
- " google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
336
- "\n",
337
- " async function convertToInteractive(key) {\n",
338
- " const element = document.querySelector('#df-b1647ad0-8c97-4561-b40f-6130d118573f');\n",
339
- " const dataTable =\n",
340
- " await google.colab.kernel.invokeFunction('convertToInteractive',\n",
341
- " [key], {});\n",
342
- " if (!dataTable) return;\n",
343
- "\n",
344
- " const docLinkHtml = 'Like what you see? Visit the ' +\n",
345
- " '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
346
- " + ' to learn more about interactive tables.';\n",
347
- " element.innerHTML = '';\n",
348
- " dataTable['output_type'] = 'display_data';\n",
349
- " await google.colab.output.renderOutput(dataTable, element);\n",
350
- " const docLink = document.createElement('div');\n",
351
- " docLink.innerHTML = docLinkHtml;\n",
352
- " element.appendChild(docLink);\n",
353
- " }\n",
354
- " </script>\n",
355
- " </div>\n",
356
- "\n",
357
- "\n",
358
- " </div>\n",
359
- " </div>\n"
360
- ],
361
- "application/vnd.google.colaboratory.intrinsic+json": {
362
- "type": "dataframe",
363
- "variable_name": "df_books",
364
- "summary": "{\n \"name\": \"df_books\",\n \"rows\": 1000,\n \"fields\": [\n {\n \"column\": \"title\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 999,\n \"samples\": [\n \"The Grownup\",\n \"Persepolis: The Story of a Childhood (Persepolis #1-2)\",\n \"Ayumi's Violin\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"price\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 14.446689669952772,\n \"min\": 10.0,\n \"max\": 59.99,\n \"num_unique_values\": 903,\n \"samples\": [\n 19.73,\n 55.65,\n 46.31\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"rating\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 5,\n \"samples\": [\n \"One\",\n \"Two\",\n \"Four\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}"
365
- }
366
- },
367
- "metadata": {},
368
- "execution_count": 45
369
- }
370
- ],
371
- "source": [
372
- "# Display the first rows\n",
373
- "df_books.head()"
374
- ]
375
- },
376
- {
377
- "cell_type": "markdown",
378
- "metadata": {
379
- "id": "p-1Pr2szaqLk"
380
- },
381
- "source": [
382
- "## **3.** 🧩 Create a meaningful connection between real & synthetic datasets"
383
- ]
384
- },
385
- {
386
- "cell_type": "markdown",
387
- "metadata": {
388
- "id": "SIaJUGIpaH4V"
389
- },
390
- "source": [
391
- "### *a. Initial setup*"
392
- ]
393
- },
394
- {
395
- "cell_type": "code",
396
- "execution_count": null,
397
- "metadata": {
398
- "id": "-gPXGcRPuV_9"
399
- },
400
- "outputs": [],
401
- "source": [
402
- "import numpy as np\n",
403
- "import random\n",
404
- "from datetime import datetime\n",
405
- "import warnings\n",
406
- "\n",
407
- "warnings.filterwarnings(\"ignore\")\n",
408
- "random.seed(2025)\n",
409
- "np.random.seed(2025)"
410
- ]
411
- },
412
- {
413
- "cell_type": "markdown",
414
- "metadata": {
415
- "id": "pY4yCoIuaQqp"
416
- },
417
- "source": [
418
- "### *b. Generate popularity scores based on rating (with some randomness) with a generate_popularity_score function*"
419
- ]
420
- },
421
- {
422
- "cell_type": "code",
423
- "execution_count": null,
424
- "metadata": {
425
- "id": "mnd5hdAbaNjz"
426
- },
427
- "outputs": [],
428
- "source": [
429
- "def generate_popularity_score(rating):\n",
430
- " base = {\"One\": 2, \"Two\": 3, \"Three\": 3, \"Four\": 4, \"Five\": 4}.get(rating, 3)\n",
431
- " trend_factor = random.choices([-1, 0, 1], weights=[1, 3, 2])[0]\n",
432
- " return int(np.clip(base + trend_factor, 1, 5))"
433
- ]
434
- },
435
- {
436
- "cell_type": "markdown",
437
- "metadata": {
438
- "id": "n4-TaNTFgPak"
439
- },
440
- "source": [
441
- "### *c. ✋🏻🛑⛔️ Run the function to create a \"popularity_score\" column from \"rating\"*"
442
- ]
443
- },
444
- {
445
- "cell_type": "code",
446
- "execution_count": null,
447
- "metadata": {
448
- "id": "V-G3OCUCgR07"
449
- },
450
- "outputs": [],
451
- "source": [
452
- "# Create the popularity_score column based on the rating column\n",
453
- "df_books[\"popularity_score\"] = df_books[\"rating\"].apply(generate_popularity_score)"
454
- ]
455
- },
456
- {
457
- "cell_type": "markdown",
458
- "metadata": {
459
- "id": "HnngRNTgacYt"
460
- },
461
- "source": [
462
- "### *d. Decide on the sentiment_label based on the popularity score with a get_sentiment function*"
463
- ]
464
- },
465
- {
466
- "cell_type": "code",
467
- "execution_count": null,
468
- "metadata": {
469
- "id": "kUtWmr8maZLZ"
470
- },
471
- "outputs": [],
472
- "source": [
473
- "def get_sentiment(popularity_score):\n",
474
- " if popularity_score <= 2:\n",
475
- " return \"negative\"\n",
476
- " elif popularity_score == 3:\n",
477
- " return \"neutral\"\n",
478
- " else:\n",
479
- " return \"positive\""
480
- ]
481
- },
482
- {
483
- "cell_type": "markdown",
484
- "metadata": {
485
- "id": "HF9F9HIzgT7Z"
486
- },
487
- "source": [
488
- "### *e. ✋🏻🛑⛔️ Run the function to create a \"sentiment_label\" column from \"popularity_score\"*"
489
- ]
490
- },
491
- {
492
- "cell_type": "code",
493
- "execution_count": null,
494
- "metadata": {
495
- "id": "tafQj8_7gYCG",
496
- "colab": {
497
- "base_uri": "https://localhost:8080/",
498
- "height": 204
499
- },
500
- "outputId": "d3842516-53a5-435b-b812-5f25259e8231"
501
- },
502
- "outputs": [
503
- {
504
- "output_type": "execute_result",
505
- "data": {
506
- "text/plain": [
507
- " title price rating popularity_score \\\n",
508
- "0 A Light in the Attic 51.77 Three 3 \n",
509
- "1 Tipping the Velvet 53.74 One 2 \n",
510
- "2 Soumission 50.10 One 2 \n",
511
- "3 Sharp Objects 47.82 Four 4 \n",
512
- "4 Sapiens: A Brief History of Humankind 54.23 Five 3 \n",
513
- "\n",
514
- " sentiment_label \n",
515
- "0 neutral \n",
516
- "1 negative \n",
517
- "2 negative \n",
518
- "3 positive \n",
519
- "4 neutral "
520
- ],
521
- "text/html": [
522
- "\n",
523
- " <div id=\"df-be660aa1-42b8-40c8-bc6f-b3bd526a4d1b\" class=\"colab-df-container\">\n",
524
- " <div>\n",
525
- "<style scoped>\n",
526
- " .dataframe tbody tr th:only-of-type {\n",
527
- " vertical-align: middle;\n",
528
- " }\n",
529
- "\n",
530
- " .dataframe tbody tr th {\n",
531
- " vertical-align: top;\n",
532
- " }\n",
533
- "\n",
534
- " .dataframe thead th {\n",
535
- " text-align: right;\n",
536
- " }\n",
537
- "</style>\n",
538
- "<table border=\"1\" class=\"dataframe\">\n",
539
- " <thead>\n",
540
- " <tr style=\"text-align: right;\">\n",
541
- " <th></th>\n",
542
- " <th>title</th>\n",
543
- " <th>price</th>\n",
544
- " <th>rating</th>\n",
545
- " <th>popularity_score</th>\n",
546
- " <th>sentiment_label</th>\n",
547
- " </tr>\n",
548
- " </thead>\n",
549
- " <tbody>\n",
550
- " <tr>\n",
551
- " <th>0</th>\n",
552
- " <td>A Light in the Attic</td>\n",
553
- " <td>51.77</td>\n",
554
- " <td>Three</td>\n",
555
- " <td>3</td>\n",
556
- " <td>neutral</td>\n",
557
- " </tr>\n",
558
- " <tr>\n",
559
- " <th>1</th>\n",
560
- " <td>Tipping the Velvet</td>\n",
561
- " <td>53.74</td>\n",
562
- " <td>One</td>\n",
563
- " <td>2</td>\n",
564
- " <td>negative</td>\n",
565
- " </tr>\n",
566
- " <tr>\n",
567
- " <th>2</th>\n",
568
- " <td>Soumission</td>\n",
569
- " <td>50.10</td>\n",
570
- " <td>One</td>\n",
571
- " <td>2</td>\n",
572
- " <td>negative</td>\n",
573
- " </tr>\n",
574
- " <tr>\n",
575
- " <th>3</th>\n",
576
- " <td>Sharp Objects</td>\n",
577
- " <td>47.82</td>\n",
578
- " <td>Four</td>\n",
579
- " <td>4</td>\n",
580
- " <td>positive</td>\n",
581
- " </tr>\n",
582
- " <tr>\n",
583
- " <th>4</th>\n",
584
- " <td>Sapiens: A Brief History of Humankind</td>\n",
585
- " <td>54.23</td>\n",
586
- " <td>Five</td>\n",
587
- " <td>3</td>\n",
588
- " <td>neutral</td>\n",
589
- " </tr>\n",
590
- " </tbody>\n",
591
- "</table>\n",
592
- "</div>\n",
593
- " <div class=\"colab-df-buttons\">\n",
594
- "\n",
595
- " <div class=\"colab-df-container\">\n",
596
- " <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-be660aa1-42b8-40c8-bc6f-b3bd526a4d1b')\"\n",
597
- " title=\"Convert this dataframe to an interactive table.\"\n",
598
- " style=\"display:none;\">\n",
599
- "\n",
600
- " <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
601
- " <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
602
- " </svg>\n",
603
- " </button>\n",
604
- "\n",
605
- " <style>\n",
606
- " .colab-df-container {\n",
607
- " display:flex;\n",
608
- " gap: 12px;\n",
609
- " }\n",
610
- "\n",
611
- " .colab-df-convert {\n",
612
- " background-color: #E8F0FE;\n",
613
- " border: none;\n",
614
- " border-radius: 50%;\n",
615
- " cursor: pointer;\n",
616
- " display: none;\n",
617
- " fill: #1967D2;\n",
618
- " height: 32px;\n",
619
- " padding: 0 0 0 0;\n",
620
- " width: 32px;\n",
621
- " }\n",
622
- "\n",
623
- " .colab-df-convert:hover {\n",
624
- " background-color: #E2EBFA;\n",
625
- " box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
626
- " fill: #174EA6;\n",
627
- " }\n",
628
- "\n",
629
- " .colab-df-buttons div {\n",
630
- " margin-bottom: 4px;\n",
631
- " }\n",
632
- "\n",
633
- " [theme=dark] .colab-df-convert {\n",
634
- " background-color: #3B4455;\n",
635
- " fill: #D2E3FC;\n",
636
- " }\n",
637
- "\n",
638
- " [theme=dark] .colab-df-convert:hover {\n",
639
- " background-color: #434B5C;\n",
640
- " box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
641
- " filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
642
- " fill: #FFFFFF;\n",
643
- " }\n",
644
- " </style>\n",
645
- "\n",
646
- " <script>\n",
647
- " const buttonEl =\n",
648
- " document.querySelector('#df-be660aa1-42b8-40c8-bc6f-b3bd526a4d1b button.colab-df-convert');\n",
649
- " buttonEl.style.display =\n",
650
- " google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
651
- "\n",
652
- " async function convertToInteractive(key) {\n",
653
- " const element = document.querySelector('#df-be660aa1-42b8-40c8-bc6f-b3bd526a4d1b');\n",
654
- " const dataTable =\n",
655
- " await google.colab.kernel.invokeFunction('convertToInteractive',\n",
656
- " [key], {});\n",
657
- " if (!dataTable) return;\n",
658
- "\n",
659
- " const docLinkHtml = 'Like what you see? Visit the ' +\n",
660
- " '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
661
- " + ' to learn more about interactive tables.';\n",
662
- " element.innerHTML = '';\n",
663
- " dataTable['output_type'] = 'display_data';\n",
664
- " await google.colab.output.renderOutput(dataTable, element);\n",
665
- " const docLink = document.createElement('div');\n",
666
- " docLink.innerHTML = docLinkHtml;\n",
667
- " element.appendChild(docLink);\n",
668
- " }\n",
669
- " </script>\n",
670
- " </div>\n",
671
- "\n",
672
- "\n",
673
- " </div>\n",
674
- " </div>\n"
675
- ],
676
- "application/vnd.google.colaboratory.intrinsic+json": {
677
- "type": "dataframe",
678
- "variable_name": "df_books",
679
- "summary": "{\n \"name\": \"df_books\",\n \"rows\": 1000,\n \"fields\": [\n {\n \"column\": \"title\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 999,\n \"samples\": [\n \"The Grownup\",\n \"Persepolis: The Story of a Childhood (Persepolis #1-2)\",\n \"Ayumi's Violin\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"price\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 14.446689669952772,\n \"min\": 10.0,\n \"max\": 59.99,\n \"num_unique_values\": 903,\n \"samples\": [\n 19.73,\n 55.65,\n 46.31\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"rating\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 5,\n \"samples\": [\n \"One\",\n \"Two\",\n \"Four\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"popularity_score\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 1,\n \"min\": 1,\n \"max\": 5,\n \"num_unique_values\": 5,\n \"samples\": [\n 2,\n 5,\n 4\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"sentiment_label\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 3,\n \"samples\": [\n \"neutral\",\n \"negative\",\n \"positive\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}"
680
- }
681
- },
682
- "metadata": {},
683
- "execution_count": 50
684
- }
685
- ],
686
- "source": [
687
- "# Create the sentiment_label column from popularity_score\n",
688
- "df_books[\"sentiment_label\"] = df_books[\"popularity_score\"].apply(get_sentiment)\n",
689
- "\n",
690
- "# Check the result\n",
691
- "df_books.head()"
692
- ]
693
- },
694
- {
695
- "cell_type": "markdown",
696
- "metadata": {
697
- "id": "T8AdKkmASq9a"
698
- },
699
- "source": [
700
- "## **4.** 📈 Generate synthetic book sales data of 18 months"
701
- ]
702
- },
703
- {
704
- "cell_type": "markdown",
705
- "metadata": {
706
- "id": "OhXbdGD5fH0c"
707
- },
708
- "source": [
709
- "### *a. Create a generate_sales_profit function that would generate sales patterns based on sentiment_label (with some randomness)*"
710
- ]
711
- },
712
- {
713
- "cell_type": "code",
714
- "execution_count": null,
715
- "metadata": {
716
- "id": "qkVhYPXGbgEn"
717
- },
718
- "outputs": [],
719
- "source": [
720
- "def generate_sales_profile(sentiment):\n",
721
- " months = pd.date_range(end=datetime.today(), periods=18, freq=\"M\")\n",
722
- "\n",
723
- " if sentiment == \"positive\":\n",
724
- " base = random.randint(200, 300)\n",
725
- " trend = np.linspace(base, base + random.randint(20, 60), len(months))\n",
726
- " elif sentiment == \"negative\":\n",
727
- " base = random.randint(20, 80)\n",
728
- " trend = np.linspace(base, base - random.randint(10, 30), len(months))\n",
729
- " else: # neutral\n",
730
- " base = random.randint(80, 160)\n",
731
- " trend = np.full(len(months), base + random.randint(-10, 10))\n",
732
- "\n",
733
- " seasonality = 10 * np.sin(np.linspace(0, 3 * np.pi, len(months)))\n",
734
- " noise = np.random.normal(0, 5, len(months))\n",
735
- " monthly_sales = np.clip(trend + seasonality + noise, a_min=0, a_max=None).astype(int)\n",
736
- "\n",
737
- " return list(zip(months.strftime(\"%Y-%m\"), monthly_sales))"
738
- ]
739
- },
740
- {
741
- "cell_type": "markdown",
742
- "metadata": {
743
- "id": "L2ak1HlcgoTe"
744
- },
745
- "source": [
746
- "### *b. Run the function as part of building sales_data*"
747
- ]
748
- },
749
- {
750
- "cell_type": "code",
751
- "execution_count": null,
752
- "metadata": {
753
- "id": "SlJ24AUafoDB"
754
- },
755
- "outputs": [],
756
- "source": [
757
- "sales_data = []\n",
758
- "for _, row in df_books.iterrows():\n",
759
- " records = generate_sales_profile(row[\"sentiment_label\"])\n",
760
- " for month, units in records:\n",
761
- " sales_data.append({\n",
762
- " \"title\": row[\"title\"],\n",
763
- " \"month\": month,\n",
764
- " \"units_sold\": units,\n",
765
- " \"sentiment_label\": row[\"sentiment_label\"]\n",
766
- " })"
767
- ]
768
- },
769
- {
770
- "cell_type": "markdown",
771
- "metadata": {
772
- "id": "4IXZKcCSgxnq"
773
- },
774
- "source": [
775
- "### *c. ✋🏻🛑⛔️ Create a df_sales DataFrame from sales_data*"
776
- ]
777
- },
778
- {
779
- "cell_type": "code",
780
- "execution_count": null,
781
- "metadata": {
782
- "id": "wcN6gtiZg-ws"
783
- },
784
- "outputs": [],
785
- "source": [
786
- "# Create the dataframe from sales_data\n",
787
- "df_sales = pd.DataFrame(sales_data)"
788
- ]
789
- },
790
- {
791
- "cell_type": "markdown",
792
- "metadata": {
793
- "id": "EhIjz9WohAmZ"
794
- },
795
- "source": [
796
- "### *d. Save df_sales as synthetic_sales_data.csv & view first few lines*"
797
- ]
798
- },
799
- {
800
- "cell_type": "code",
801
- "execution_count": null,
802
- "metadata": {
803
- "colab": {
804
- "base_uri": "https://localhost:8080/"
805
- },
806
- "id": "MzbZvLcAhGaH",
807
- "outputId": "c3cc8b7a-df5f-40da-b1eb-3f00144d05fa"
808
- },
809
- "outputs": [
810
- {
811
- "output_type": "stream",
812
- "name": "stdout",
813
- "text": [
814
- " title month units_sold sentiment_label\n",
815
- "0 A Light in the Attic 2024-09 100 neutral\n",
816
- "1 A Light in the Attic 2024-10 109 neutral\n",
817
- "2 A Light in the Attic 2024-11 102 neutral\n",
818
- "3 A Light in the Attic 2024-12 107 neutral\n",
819
- "4 A Light in the Attic 2025-01 108 neutral\n"
820
- ]
821
- }
822
- ],
823
- "source": [
824
- "df_sales.to_csv(\"synthetic_sales_data.csv\", index=False)\n",
825
- "\n",
826
- "print(df_sales.head())"
827
- ]
828
- },
829
- {
830
- "cell_type": "markdown",
831
- "metadata": {
832
- "id": "7g9gqBgQMtJn"
833
- },
834
- "source": [
835
- "## **5.** 🎯 Generate synthetic customer reviews"
836
- ]
837
- },
838
- {
839
- "cell_type": "markdown",
840
- "metadata": {
841
- "id": "Gi4y9M9KuDWx"
842
- },
843
- "source": [
844
- "### *a. ✋🏻🛑⛔️ Ask ChatGPT to create a list of 50 distinct generic book review texts for the sentiment labels \"positive\", \"neutral\", and \"negative\" called synthetic_reviews_by_sentiment*"
845
- ]
846
- },
847
- {
848
- "cell_type": "code",
849
- "source": [
850
- "synthetic_reviews_by_sentiment = {\n",
851
- " \"positive\": [\n",
852
- " \"A compelling and heartwarming read that stayed with me long after I finished.\",\n",
853
- " \"Brilliantly written! The characters were unforgettable and the plot was engaging.\",\n",
854
- " \"One of the best books I've read this year — inspiring and emotionally rich.\",\n",
855
- " \"A beautifully crafted story with vivid characters and a satisfying ending.\",\n",
856
- " \"An uplifting book that kept me turning the pages late into the night.\",\n",
857
- " \"Thoughtful, engaging, and wonderfully written from start to finish.\",\n",
858
- " \"An excellent read that blends emotion, humor, and depth perfectly.\",\n",
859
- " \"I loved every chapter — the storytelling was simply outstanding.\",\n",
860
- " \"A memorable story that left a lasting impression.\",\n",
861
- " \"The narrative flowed beautifully and kept me completely invested.\",\n",
862
- " \"A truly enjoyable book that exceeded my expectations.\",\n",
863
- " \"The author created a world I didn’t want to leave.\",\n",
864
- " \"Smart, engaging, and deeply satisfying to read.\",\n",
865
- " \"A powerful story that was both entertaining and meaningful.\",\n",
866
- " \"An inspiring read that kept me hooked throughout.\",\n",
867
- " \"A delightful surprise — captivating from beginning to end.\",\n",
868
- " \"Fantastic storytelling with characters that felt real.\",\n",
869
- " \"A wonderfully immersive and emotionally rewarding read.\",\n",
870
- " \"The pacing and character development were excellent.\",\n",
871
- " \"A charming and thoughtful story that was hard to put down.\",\n",
872
- " \"A rich narrative filled with emotion and insight.\",\n",
873
- " \"An incredibly engaging book that I would gladly read again.\",\n",
874
- " \"The writing style was smooth and very enjoyable.\",\n",
875
- " \"A standout book with strong characters and an engaging plot.\",\n",
876
- " \"An unforgettable journey that I thoroughly enjoyed.\",\n",
877
- " \"A brilliant story with layers of emotion and meaning.\",\n",
878
- " \"One of those books you wish wouldn’t end.\",\n",
879
- " \"A touching and beautifully written novel.\",\n",
880
- " \"The author’s storytelling skills truly shine here.\",\n",
881
- " \"A captivating book that delivered on every level.\",\n",
882
- " \"A fantastic read with a deeply satisfying conclusion.\",\n",
883
- " \"The story was compelling and emotionally powerful.\",\n",
884
- " \"An inspiring narrative that left me feeling uplifted.\",\n",
885
- " \"A highly enjoyable book with great storytelling.\",\n",
886
- " \"A refreshing and engaging reading experience.\",\n",
887
- " \"A story that was both entertaining and thoughtful.\",\n",
888
- " \"A beautifully paced novel with memorable moments.\",\n",
889
- " \"An excellent book that kept me fully engaged.\",\n",
890
- " \"The writing was vivid and the story was captivating.\",\n",
891
- " \"A remarkable book that stayed with me afterwards.\",\n",
892
- " \"A joyful reading experience from beginning to end.\",\n",
893
- " \"The characters and story felt authentic and meaningful.\",\n",
894
- " \"An engaging and heartfelt novel.\",\n",
895
- " \"A book that balances emotion and storytelling perfectly.\",\n",
896
- " \"A delightful and rewarding read.\",\n",
897
- " \"The author crafted an impressive and memorable story.\",\n",
898
- " \"A fantastic book that I would highly recommend.\",\n",
899
- " \"A powerful narrative with strong emotional depth.\",\n",
900
- " \"A beautifully told story that resonated with me.\",\n",
901
- " \"An exceptional book that truly stood out.\"\n",
902
- " ],\n",
903
- "\n",
904
- " \"neutral\": [\n",
905
- " \"An average book — not great, but not bad either.\",\n",
906
- " \"Some parts really stood out, others felt a bit flat.\",\n",
907
- " \"It was okay overall. A decent way to pass the time.\",\n",
908
- " \"A fairly standard read without many surprises.\",\n",
909
- " \"The story had some interesting moments but nothing remarkable.\",\n",
910
- " \"Not particularly memorable, but still readable.\",\n",
911
- " \"A mixed experience with both good and weaker sections.\",\n",
912
- " \"The book was enjoyable in parts but inconsistent overall.\",\n",
913
- " \"A decent story that didn’t quite reach its potential.\",\n",
914
- " \"The pacing was uneven, though the idea was interesting.\",\n",
915
- " \"Some chapters were engaging while others dragged.\",\n",
916
- " \"An average novel with a few bright spots.\",\n",
917
- " \"Not bad, but it didn’t leave a strong impression.\",\n",
918
- " \"A straightforward story with predictable elements.\",\n",
919
- " \"The concept was good but the execution was average.\",\n",
920
- " \"A readable book that lacked a strong impact.\",\n",
921
- " \"It held my attention occasionally but not consistently.\",\n",
922
- " \"The book was fine, though not particularly exciting.\",\n",
923
- " \"Some good ideas, but the storytelling felt uneven.\",\n",
924
- " \"A moderate reading experience overall.\",\n",
925
- " \"The characters were interesting but not very deep.\",\n",
926
- " \"A passable read with a few enjoyable moments.\",\n",
927
- " \"Neither impressive nor disappointing.\",\n",
928
- " \"The book had potential but felt somewhat ordinary.\",\n",
929
- " \"A simple story that was easy to follow.\",\n",
930
- " \"An acceptable read, though not especially memorable.\",\n",
931
- " \"It had its moments but didn’t fully engage me.\",\n",
932
- " \"A fairly predictable narrative.\",\n",
933
- " \"A balanced mix of interesting and dull moments.\",\n",
934
- " \"Not a bad book, just not very distinctive.\",\n",
935
- " \"An okay read for a quiet afternoon.\",\n",
936
- " \"Some interesting scenes but overall average.\",\n",
937
- " \"The story worked, though it lacked excitement.\",\n",
938
- " \"A decent attempt but not particularly standout.\",\n",
939
- " \"The pacing and story were fairly standard.\",\n",
940
- " \"An ordinary reading experience.\",\n",
941
- " \"Enjoyable at times but inconsistent.\",\n",
942
- " \"The writing was fine but not remarkable.\",\n",
943
- " \"A serviceable story with few surprises.\",\n",
944
- " \"A mild reading experience overall.\",\n",
945
- " \"Interesting concept but average delivery.\",\n",
946
- " \"The book was pleasant but not very engaging.\",\n",
947
- " \"It kept my attention occasionally.\",\n",
948
- " \"A straightforward but somewhat forgettable read.\",\n",
949
- " \"A modestly entertaining story.\",\n",
950
- " \"The narrative was steady but not gripping.\",\n",
951
- " \"A readable book that didn’t stand out.\",\n",
952
- " \"An average novel with moderate appeal.\",\n",
953
- " \"Decent but not particularly compelling.\",\n",
954
- " \"Overall, an okay book.\"\n",
955
- " ],\n",
956
- "\n",
957
- " \"negative\": [\n",
958
- " \"I struggled to get through this one — it just didn’t grab me.\",\n",
959
- " \"The plot was confusing and the characters felt underdeveloped.\",\n",
960
- " \"Disappointing. I had high hopes, but they weren't met.\",\n",
961
- " \"The story felt slow and lacked excitement.\",\n",
962
- " \"I found it difficult to stay interested.\",\n",
963
- " \"The characters were hard to connect with.\",\n",
964
- " \"The pacing dragged and the story never really picked up.\",\n",
965
- " \"A frustrating read that didn’t deliver.\",\n",
966
- " \"The narrative felt disorganized and unclear.\",\n",
967
- " \"I kept waiting for it to improve, but it never did.\",\n",
968
- " \"The writing style didn’t work for me.\",\n",
969
- " \"A forgettable story with little impact.\",\n",
970
- " \"The plot felt predictable and uninspired.\",\n",
971
- " \"I struggled to stay engaged with the story.\",\n",
972
- " \"The characters lacked depth and realism.\",\n",
973
- " \"The story never really found its direction.\",\n",
974
- " \"A disappointing reading experience.\",\n",
975
- " \"The pacing was uneven and often slow.\",\n",
976
- " \"The book felt longer than it needed to be.\",\n",
977
- " \"A confusing plot that was hard to follow.\",\n",
978
- " \"The writing felt flat and uninspired.\",\n",
979
- " \"The story lacked energy and excitement.\",\n",
980
- " \"I couldn’t connect with the narrative.\",\n",
981
- " \"The book didn’t live up to expectations.\",\n",
982
- " \"The characters were forgettable.\",\n",
983
- " \"The story felt repetitive and dull.\",\n",
984
- " \"It was difficult to stay invested in the plot.\",\n",
985
- " \"The book lacked originality.\",\n",
986
- " \"A weak storyline that never developed properly.\",\n",
987
- " \"The narrative felt unfocused.\",\n",
988
- " \"A disappointing and uneven read.\",\n",
989
- " \"The book failed to hold my attention.\",\n",
990
- " \"The writing style made it hard to enjoy.\",\n",
991
- " \"The pacing slowed the story down too much.\",\n",
992
- " \"The plot felt shallow and predictable.\",\n",
993
- " \"I expected more depth from the story.\",\n",
994
- " \"The book lacked emotional impact.\",\n",
995
- " \"A frustrating and slow reading experience.\",\n",
996
- " \"The narrative never became engaging.\",\n",
997
- " \"The characters felt unrealistic.\",\n",
998
- " \"The story was difficult to follow.\",\n",
999
- " \"The writing lacked clarity.\",\n",
1000
- " \"A dull and unremarkable book.\",\n",
1001
- " \"The storyline felt weak and incomplete.\",\n",
1002
- " \"The book struggled to maintain interest.\",\n",
1003
- " \"A disappointing attempt at storytelling.\",\n",
1004
- " \"The pacing and plot didn’t work well together.\",\n",
1005
- " \"The story felt rushed in some places and slow in others.\",\n",
1006
- " \"The book ultimately failed to impress.\",\n",
1007
- " \"Not an enjoyable reading experience.\"\n",
1008
- " ]\n",
1009
- "}"
1010
- ],
1011
- "metadata": {
1012
- "id": "W8woph8MLJoV"
1013
- },
1014
- "execution_count": null,
1015
- "outputs": []
1016
- },
1017
- {
1018
- "cell_type": "markdown",
1019
- "metadata": {
1020
- "id": "fQhfVaDmuULT"
1021
- },
1022
- "source": [
1023
- "### *b. Generate 10 reviews per book using random sampling from the corresponding 50*"
1024
- ]
1025
- },
1026
- {
1027
- "cell_type": "code",
1028
- "execution_count": null,
1029
- "metadata": {
1030
- "id": "l2SRc3PjuTGM"
1031
- },
1032
- "outputs": [],
1033
- "source": [
1034
- "review_rows = []\n",
1035
- "for _, row in df_books.iterrows():\n",
1036
- " title = row['title']\n",
1037
- " sentiment_label = row['sentiment_label']\n",
1038
- " review_pool = synthetic_reviews_by_sentiment[sentiment_label]\n",
1039
- " sampled_reviews = random.sample(review_pool, 10)\n",
1040
- " for review_text in sampled_reviews:\n",
1041
- " review_rows.append({\n",
1042
- " \"title\": title,\n",
1043
- " \"sentiment_label\": sentiment_label,\n",
1044
- " \"review_text\": review_text,\n",
1045
- " \"rating\": row['rating'],\n",
1046
- " \"popularity_score\": row['popularity_score']\n",
1047
- " })"
1048
- ]
1049
- },
1050
- {
1051
- "cell_type": "markdown",
1052
- "metadata": {
1053
- "id": "bmJMXF-Bukdm"
1054
- },
1055
- "source": [
1056
- "### *c. Create the final dataframe df_reviews & save it as synthetic_book_reviews.csv*"
1057
- ]
1058
- },
1059
- {
1060
- "cell_type": "code",
1061
- "execution_count": null,
1062
- "metadata": {
1063
- "id": "ZUKUqZsuumsp"
1064
- },
1065
- "outputs": [],
1066
- "source": [
1067
- "df_reviews = pd.DataFrame(review_rows)\n",
1068
- "df_reviews.to_csv(\"synthetic_book_reviews.csv\", index=False)"
1069
- ]
1070
- },
1071
- {
1072
- "cell_type": "markdown",
1073
- "source": [
1074
- "### *c. inputs for R*"
1075
- ],
1076
- "metadata": {
1077
- "id": "_602pYUS3gY5"
1078
- }
1079
- },
1080
- {
1081
- "cell_type": "code",
1082
- "execution_count": null,
1083
- "metadata": {
1084
- "colab": {
1085
- "base_uri": "https://localhost:8080/"
1086
- },
1087
- "id": "3946e521",
1088
- "outputId": "cad60205-7605-4d60-ffba-68a563a46994"
1089
- },
1090
- "outputs": [
1091
- {
1092
- "output_type": "stream",
1093
- "name": "stdout",
1094
- "text": [
1095
- "✅ Wrote synthetic_title_level_features.csv\n",
1096
- "✅ Wrote synthetic_monthly_revenue_series.csv\n"
1097
- ]
1098
- }
1099
- ],
1100
- "source": [
1101
- "import numpy as np\n",
1102
- "\n",
1103
- "def _safe_num(s):\n",
1104
- " return pd.to_numeric(\n",
1105
- " pd.Series(s).astype(str).str.replace(r\"[^0-9.]\", \"\", regex=True),\n",
1106
- " errors=\"coerce\"\n",
1107
- " )\n",
1108
- "\n",
1109
- "# --- Clean book metadata (price/rating) ---\n",
1110
- "df_books_r = df_books.copy()\n",
1111
- "if \"price\" in df_books_r.columns:\n",
1112
- " df_books_r[\"price\"] = _safe_num(df_books_r[\"price\"])\n",
1113
- "if \"rating\" in df_books_r.columns:\n",
1114
- " df_books_r[\"rating\"] = _safe_num(df_books_r[\"rating\"])\n",
1115
- "\n",
1116
- "df_books_r[\"title\"] = df_books_r[\"title\"].astype(str).str.strip()\n",
1117
- "\n",
1118
- "# --- Clean sales ---\n",
1119
- "df_sales_r = df_sales.copy()\n",
1120
- "df_sales_r[\"title\"] = df_sales_r[\"title\"].astype(str).str.strip()\n",
1121
- "df_sales_r[\"month\"] = pd.to_datetime(df_sales_r[\"month\"], errors=\"coerce\")\n",
1122
- "df_sales_r[\"units_sold\"] = _safe_num(df_sales_r[\"units_sold\"])\n",
1123
- "\n",
1124
- "# --- Clean reviews ---\n",
1125
- "df_reviews_r = df_reviews.copy()\n",
1126
- "df_reviews_r[\"title\"] = df_reviews_r[\"title\"].astype(str).str.strip()\n",
1127
- "df_reviews_r[\"sentiment_label\"] = df_reviews_r[\"sentiment_label\"].astype(str).str.lower().str.strip()\n",
1128
- "if \"rating\" in df_reviews_r.columns:\n",
1129
- " df_reviews_r[\"rating\"] = _safe_num(df_reviews_r[\"rating\"])\n",
1130
- "if \"popularity_score\" in df_reviews_r.columns:\n",
1131
- " df_reviews_r[\"popularity_score\"] = _safe_num(df_reviews_r[\"popularity_score\"])\n",
1132
- "\n",
1133
- "# --- Sentiment shares per title (from reviews) ---\n",
1134
- "sent_counts = (\n",
1135
- " df_reviews_r.groupby([\"title\", \"sentiment_label\"])\n",
1136
- " .size()\n",
1137
- " .unstack(fill_value=0)\n",
1138
- ")\n",
1139
- "for lab in [\"positive\", \"neutral\", \"negative\"]:\n",
1140
- " if lab not in sent_counts.columns:\n",
1141
- " sent_counts[lab] = 0\n",
1142
- "\n",
1143
- "sent_counts[\"total_reviews\"] = sent_counts[[\"positive\", \"neutral\", \"negative\"]].sum(axis=1)\n",
1144
- "den = sent_counts[\"total_reviews\"].replace(0, np.nan)\n",
1145
- "sent_counts[\"share_positive\"] = sent_counts[\"positive\"] / den\n",
1146
- "sent_counts[\"share_neutral\"] = sent_counts[\"neutral\"] / den\n",
1147
- "sent_counts[\"share_negative\"] = sent_counts[\"negative\"] / den\n",
1148
- "sent_counts = sent_counts.reset_index()\n",
1149
- "\n",
1150
- "# --- Sales aggregation per title ---\n",
1151
- "sales_by_title = (\n",
1152
- " df_sales_r.dropna(subset=[\"title\"])\n",
1153
- " .groupby(\"title\", as_index=False)\n",
1154
- " .agg(\n",
1155
- " months_observed=(\"month\", \"nunique\"),\n",
1156
- " avg_units_sold=(\"units_sold\", \"mean\"),\n",
1157
- " total_units_sold=(\"units_sold\", \"sum\"),\n",
1158
- " )\n",
1159
- ")\n",
1160
- "\n",
1161
- "# --- Title-level features (join sales + books + sentiment) ---\n",
1162
- "df_title = (\n",
1163
- " sales_by_title\n",
1164
- " .merge(df_books_r[[\"title\", \"price\", \"rating\"]], on=\"title\", how=\"left\")\n",
1165
- " .merge(sent_counts[[\"title\", \"share_positive\", \"share_neutral\", \"share_negative\", \"total_reviews\"]],\n",
1166
- " on=\"title\", how=\"left\")\n",
1167
- ")\n",
1168
- "\n",
1169
- "df_title[\"avg_revenue\"] = df_title[\"avg_units_sold\"] * df_title[\"price\"]\n",
1170
- "df_title[\"total_revenue\"] = df_title[\"total_units_sold\"] * df_title[\"price\"]\n",
1171
- "\n",
1172
- "df_title.to_csv(\"synthetic_title_level_features.csv\", index=False)\n",
1173
- "print(\"✅ Wrote synthetic_title_level_features.csv\")\n",
1174
- "\n",
1175
- "# --- Monthly revenue series (proxy: units_sold * price) ---\n",
1176
- "monthly_rev = (\n",
1177
- " df_sales_r.merge(df_books_r[[\"title\", \"price\"]], on=\"title\", how=\"left\")\n",
1178
- ")\n",
1179
- "monthly_rev[\"revenue\"] = monthly_rev[\"units_sold\"] * monthly_rev[\"price\"]\n",
1180
- "\n",
1181
- "df_monthly = (\n",
1182
- " monthly_rev.dropna(subset=[\"month\"])\n",
1183
- " .groupby(\"month\", as_index=False)[\"revenue\"]\n",
1184
- " .sum()\n",
1185
- " .rename(columns={\"revenue\": \"total_revenue\"})\n",
1186
- " .sort_values(\"month\")\n",
1187
- ")\n",
1188
- "# if revenue is all NA (e.g., missing price), fallback to units_sold as a teaching proxy\n",
1189
- "if df_monthly[\"total_revenue\"].notna().sum() == 0:\n",
1190
- " df_monthly = (\n",
1191
- " df_sales_r.dropna(subset=[\"month\"])\n",
1192
- " .groupby(\"month\", as_index=False)[\"units_sold\"]\n",
1193
- " .sum()\n",
1194
- " .rename(columns={\"units_sold\": \"total_revenue\"})\n",
1195
- " .sort_values(\"month\")\n",
1196
- " )\n",
1197
- "\n",
1198
- "df_monthly[\"month\"] = pd.to_datetime(df_monthly[\"month\"], errors=\"coerce\").dt.strftime(\"%Y-%m-%d\")\n",
1199
- "df_monthly.to_csv(\"synthetic_monthly_revenue_series.csv\", index=False)\n",
1200
- "print(\"✅ Wrote synthetic_monthly_revenue_series.csv\")\n"
1201
- ]
1202
- },
1203
- {
1204
- "cell_type": "markdown",
1205
- "metadata": {
1206
- "id": "RYvGyVfXuo54"
1207
- },
1208
- "source": [
1209
- "### *d. ✋🏻🛑⛔️ View the first few lines*"
1210
- ]
1211
- },
1212
- {
1213
- "cell_type": "code",
1214
- "execution_count": null,
1215
- "metadata": {
1216
- "colab": {
1217
- "base_uri": "https://localhost:8080/",
1218
- "height": 204
1219
- },
1220
- "id": "xfE8NMqOurKo",
1221
- "outputId": "c50b9f1e-3aed-49e9-f9ae-f2441bda8f30"
1222
- },
1223
- "outputs": [
1224
- {
1225
- "output_type": "execute_result",
1226
- "data": {
1227
- "text/plain": [
1228
- " title sentiment_label \\\n",
1229
- "0 A Light in the Attic neutral \n",
1230
- "1 A Light in the Attic neutral \n",
1231
- "2 A Light in the Attic neutral \n",
1232
- "3 A Light in the Attic neutral \n",
1233
- "4 A Light in the Attic neutral \n",
1234
- "\n",
1235
- " review_text rating popularity_score \n",
1236
- "0 A readable book that didn’t stand out. Three 3 \n",
1237
- "1 Neither impressive nor disappointing. Three 3 \n",
1238
- "2 The pacing and story were fairly standard. Three 3 \n",
1239
- "3 A decent story that didn’t quite reach its pot... Three 3 \n",
1240
- "4 A modestly entertaining story. Three 3 "
1241
- ],
1242
- "text/html": [
1243
- "\n",
1244
- " <div id=\"df-6e128220-4516-4f55-8e04-ba69ac6a190d\" class=\"colab-df-container\">\n",
1245
- " <div>\n",
1246
- "<style scoped>\n",
1247
- " .dataframe tbody tr th:only-of-type {\n",
1248
- " vertical-align: middle;\n",
1249
- " }\n",
1250
- "\n",
1251
- " .dataframe tbody tr th {\n",
1252
- " vertical-align: top;\n",
1253
- " }\n",
1254
- "\n",
1255
- " .dataframe thead th {\n",
1256
- " text-align: right;\n",
1257
- " }\n",
1258
- "</style>\n",
1259
- "<table border=\"1\" class=\"dataframe\">\n",
1260
- " <thead>\n",
1261
- " <tr style=\"text-align: right;\">\n",
1262
- " <th></th>\n",
1263
- " <th>title</th>\n",
1264
- " <th>sentiment_label</th>\n",
1265
- " <th>review_text</th>\n",
1266
- " <th>rating</th>\n",
1267
- " <th>popularity_score</th>\n",
1268
- " </tr>\n",
1269
- " </thead>\n",
1270
- " <tbody>\n",
1271
- " <tr>\n",
1272
- " <th>0</th>\n",
1273
- " <td>A Light in the Attic</td>\n",
1274
- " <td>neutral</td>\n",
1275
- " <td>A readable book that didn’t stand out.</td>\n",
1276
- " <td>Three</td>\n",
1277
- " <td>3</td>\n",
1278
- " </tr>\n",
1279
- " <tr>\n",
1280
- " <th>1</th>\n",
1281
- " <td>A Light in the Attic</td>\n",
1282
- " <td>neutral</td>\n",
1283
- " <td>Neither impressive nor disappointing.</td>\n",
1284
- " <td>Three</td>\n",
1285
- " <td>3</td>\n",
1286
- " </tr>\n",
1287
- " <tr>\n",
1288
- " <th>2</th>\n",
1289
- " <td>A Light in the Attic</td>\n",
1290
- " <td>neutral</td>\n",
1291
- " <td>The pacing and story were fairly standard.</td>\n",
1292
- " <td>Three</td>\n",
1293
- " <td>3</td>\n",
1294
- " </tr>\n",
1295
- " <tr>\n",
1296
- " <th>3</th>\n",
1297
- " <td>A Light in the Attic</td>\n",
1298
- " <td>neutral</td>\n",
1299
- " <td>A decent story that didn’t quite reach its pot...</td>\n",
1300
- " <td>Three</td>\n",
1301
- " <td>3</td>\n",
1302
- " </tr>\n",
1303
- " <tr>\n",
1304
- " <th>4</th>\n",
1305
- " <td>A Light in the Attic</td>\n",
1306
- " <td>neutral</td>\n",
1307
- " <td>A modestly entertaining story.</td>\n",
1308
- " <td>Three</td>\n",
1309
- " <td>3</td>\n",
1310
- " </tr>\n",
1311
- " </tbody>\n",
1312
- "</table>\n",
1313
- "</div>\n",
1314
- " <div class=\"colab-df-buttons\">\n",
1315
- "\n",
1316
- " <div class=\"colab-df-container\">\n",
1317
- " <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-6e128220-4516-4f55-8e04-ba69ac6a190d')\"\n",
1318
- " title=\"Convert this dataframe to an interactive table.\"\n",
1319
- " style=\"display:none;\">\n",
1320
- "\n",
1321
- " <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
1322
- " <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
1323
- " </svg>\n",
1324
- " </button>\n",
1325
- "\n",
1326
- " <style>\n",
1327
- " .colab-df-container {\n",
1328
- " display:flex;\n",
1329
- " gap: 12px;\n",
1330
- " }\n",
1331
- "\n",
1332
- " .colab-df-convert {\n",
1333
- " background-color: #E8F0FE;\n",
1334
- " border: none;\n",
1335
- " border-radius: 50%;\n",
1336
- " cursor: pointer;\n",
1337
- " display: none;\n",
1338
- " fill: #1967D2;\n",
1339
- " height: 32px;\n",
1340
- " padding: 0 0 0 0;\n",
1341
- " width: 32px;\n",
1342
- " }\n",
1343
- "\n",
1344
- " .colab-df-convert:hover {\n",
1345
- " background-color: #E2EBFA;\n",
1346
- " box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
1347
- " fill: #174EA6;\n",
1348
- " }\n",
1349
- "\n",
1350
- " .colab-df-buttons div {\n",
1351
- " margin-bottom: 4px;\n",
1352
- " }\n",
1353
- "\n",
1354
- " [theme=dark] .colab-df-convert {\n",
1355
- " background-color: #3B4455;\n",
1356
- " fill: #D2E3FC;\n",
1357
- " }\n",
1358
- "\n",
1359
- " [theme=dark] .colab-df-convert:hover {\n",
1360
- " background-color: #434B5C;\n",
1361
- " box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
1362
- " filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
1363
- " fill: #FFFFFF;\n",
1364
- " }\n",
1365
- " </style>\n",
1366
- "\n",
1367
- " <script>\n",
1368
- " const buttonEl =\n",
1369
- " document.querySelector('#df-6e128220-4516-4f55-8e04-ba69ac6a190d button.colab-df-convert');\n",
1370
- " buttonEl.style.display =\n",
1371
- " google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
1372
- "\n",
1373
- " async function convertToInteractive(key) {\n",
1374
- " const element = document.querySelector('#df-6e128220-4516-4f55-8e04-ba69ac6a190d');\n",
1375
- " const dataTable =\n",
1376
- " await google.colab.kernel.invokeFunction('convertToInteractive',\n",
1377
- " [key], {});\n",
1378
- " if (!dataTable) return;\n",
1379
- "\n",
1380
- " const docLinkHtml = 'Like what you see? Visit the ' +\n",
1381
- " '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
1382
- " + ' to learn more about interactive tables.';\n",
1383
- " element.innerHTML = '';\n",
1384
- " dataTable['output_type'] = 'display_data';\n",
1385
- " await google.colab.output.renderOutput(dataTable, element);\n",
1386
- " const docLink = document.createElement('div');\n",
1387
- " docLink.innerHTML = docLinkHtml;\n",
1388
- " element.appendChild(docLink);\n",
1389
- " }\n",
1390
- " </script>\n",
1391
- " </div>\n",
1392
- "\n",
1393
- "\n",
1394
- " </div>\n",
1395
- " </div>\n"
1396
- ],
1397
- "application/vnd.google.colaboratory.intrinsic+json": {
1398
- "type": "dataframe",
1399
- "variable_name": "df_reviews",
1400
- "summary": "{\n \"name\": \"df_reviews\",\n \"rows\": 10000,\n \"fields\": [\n {\n \"column\": \"title\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 999,\n \"samples\": [\n \"The Grownup\",\n \"Persepolis: The Story of a Childhood (Persepolis #1-2)\",\n \"Ayumi's Violin\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"sentiment_label\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 3,\n \"samples\": [\n \"neutral\",\n \"negative\",\n \"positive\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"review_text\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 150,\n \"samples\": [\n \"A rich narrative filled with emotion and insight.\",\n \"The characters lacked depth and realism.\",\n \"A fantastic read with a deeply satisfying conclusion.\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"rating\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 5,\n \"samples\": [\n \"One\",\n \"Two\",\n \"Four\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"popularity_score\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 1,\n \"min\": 1,\n \"max\": 5,\n \"num_unique_values\": 5,\n \"samples\": [\n 2,\n 5,\n 4\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}"
1401
- }
1402
- },
1403
- "metadata": {},
1404
- "execution_count": 59
1405
- }
1406
- ],
1407
- "source": [
1408
- "df_reviews.head()"
1409
- ]
1410
- }
1411
- ],
1412
- "metadata": {
1413
- "colab": {
1414
- "provenance": []
1415
- },
1416
- "kernelspec": {
1417
- "display_name": "Python 3",
1418
- "name": "python3"
1419
- },
1420
- "language_info": {
1421
- "name": "python"
1422
- }
1423
- },
1424
- "nbformat": 4,
1425
- "nbformat_minor": 0
1426
- }