Files reorganized

Files reorganized according to topic, more details to example.ipynb were added

EXAMPLES/Example.ipynb

@@ -1,49 +1,115 @@
 {
   "cells": [
+    {
+      "cell_type": "markdown",
+      "source": [
+        "# Introduction\n",
+        "\n",
+        "NexDatawork is an AI data agent for data engineering and analytics without writing code.\n",
+        "\n",
+        "## Prerequisites\n",
+        "\n",
+        " - Node.js\n",
+        " - Supabase\n",
+        " - OpenAI\n",
+        "\n",
+        "\n",
+        " Before starting your work install all the required tools:"
+      ],
+      "metadata": {
+        "id": "w8J6iWjvjZJd"
+      },
+      "id": "w8J6iWjvjZJd"
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "!pip install langchain_openai\n",
+        "!pip install langchain\n",
+        "!pip install langchain_experimental"
+      ],
+      "metadata": {
+        "id": "L0KlJIwcdCed"
+      },
+      "execution_count": 1,
+      "outputs": [],
+      "id": "L0KlJIwcdCed"
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "Import all the required libraries"
+      ],
+      "metadata": {
+        "id": "uiqjnn4LkaGJ"
+      },
+      "id": "uiqjnn4LkaGJ"
+    },
     {
       "cell_type": "code",
       "execution_count": null,
-      "id": "48250b17-aada-4838-8fe9-843fe970b904",
       "metadata": {
-        "id": "48250b17-aada-4838-8fe9-843fe970b904"
+        "id": "u3Py4qe6PlP7"
       },
       "outputs": [],
       "source": [
         "import os\n",
         "import pandas as pd\n",
-        "from IPython.display import Markdown, HTML, display"
-      ]
+        "from IPython.display import Markdown, HTML, display\n",
+        "from langchain_openai import AzureChatOpenAI\n",
+        "from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler\n",
+        "from langchain.agents.agent_types import AgentType\n",
+        "from langchain_experimental.agents.agent_toolkits import create_pandas_dataframe_agent\n",
+        "import gradio as gr\n",
+        "import pandas as pd\n",
+        "import io\n",
+        "import contextlib"
+      ],
+      "id": "u3Py4qe6PlP7"
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "To access AzureOpenAI models you'll need to create an Azure account, create a deployment of an Azure OpenAI model, get the name and endpoint for your deployment, get an Azure OpenAI API key and install the langchain-openai integration package.\n",
+        "\n",
+        "Replace the placeholders with the actual values."
+      ],
+      "metadata": {
+        "id": "F5YuW2LyosRX"
+      },
+      "id": "F5YuW2LyosRX"
     },
     {
       "cell_type": "code",
-      "execution_count": 1,
-      "id": "146be3c7-90df-4fbe-bff6-00166f3d61d2",
+      "execution_count": null,
       "metadata": {
-        "id": "146be3c7-90df-4fbe-bff6-00166f3d61d2"
+        "id": "5ldGnv_jPlP8"
       },
       "outputs": [],
       "source": [
-        "import os\n",
-        "\n",
-        "# Replace with your actual values\n",
         "os.environ[\"AZURE_OPENAI_ENDPOINT\"] = \"INSERT THE OPENAI ENDPOINT\"\n",
-        "os.environ[\"AZURE_OPENAI_API_KEY\"] = \"INSERT YOUR OPENAI API KEY\"\n"
-      ]
+        "os.environ[\"AZURE_OPENAI_API_KEY\"] = \"INSERT YOUR OPENAI API KEY"
+      ],
+      "id": "5ldGnv_jPlP8"
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "To set up the Azure OpenAI model choose the name for ```AZURE_DEPLOYMENT_NAME``` and insert ```AZURE_API_VERSION``` (the latest supported version can be found here: https://learn.microsoft.com/en-us/azure/ai-services/openai/reference)."
+      ],
+      "metadata": {
+        "id": "TWDsP2d0qP5q"
+      },
+      "id": "TWDsP2d0qP5q"
     },
     {
       "cell_type": "code",
       "execution_count": null,
-      "id": "f5e1b596-4568-4078-ae14-b20d25cba62b",
       "metadata": {
         "id": "f5e1b596-4568-4078-ae14-b20d25cba62b"
       },
       "outputs": [],
       "source": [
-        "# 2nd Cell: Azure OpenAI setup\n",
-        "import os\n",
-        "from langchain_openai import AzureChatOpenAI\n",
-        "from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler\n",
-        "\n",
         "# Load your Azure environment variables\n",
         "AZURE_OPENAI_ENDPOINT = os.getenv(\"AZURE_OPENAI_ENDPOINT\")\n",
         "AZURE_DEPLOYMENT_NAME = \"gpt-4.1\"  # 👈 Change if needed\n",
@@ -57,32 +123,22 @@
         "    streaming=True,\n",
         "    callbacks=[StreamingStdOutCallbackHandler()],\n",
         ")\n"
-      ]
+      ],
+      "id": "f5e1b596-4568-4078-ae14-b20d25cba62b"
     },
     {
-      "cell_type": "code",
-      "execution_count": 4,
-      "id": "789b46d9-2189-4d3c-8f77-61b4675bf950",
+      "cell_type": "markdown",
+      "source": [
+        "Here you can create a prompt for your model."
+      ],
       "metadata": {
-        "id": "789b46d9-2189-4d3c-8f77-61b4675bf950"
+        "id": "9cMbxsJuyMPb"
       },
-      "outputs": [],
+      "id": "9cMbxsJuyMPb"
+    },
+    {
+      "cell_type": "code",
       "source": [
-        "# --- Setup ---\n",
-        "import os\n",
-        "import gradio as gr\n",
-        "import pandas as pd\n",
-        "import io\n",
-        "import contextlib\n",
-        "\n",
-        "from langchain.agents.agent_types import AgentType\n",
-        "from langchain_experimental.agents.agent_toolkits import create_pandas_dataframe_agent\n",
-        "\n",
-        "# Replace this with your actual LLM setup\n",
-        "# Example:\n",
-        "# from langchain_openai import AzureChatOpenAI\n",
-        "# model = AzureChatOpenAI(...)\n",
-        "\n",
         "# Prompt\n",
         "CSV_PROMPT_PREFIX = \"\"\"\n",
         "Set pandas to show all columns.\n",
@@ -110,6 +166,58 @@
         "Please provide only the Python code required to perform the action, and nothing else until the final Markdown output.\n",
         "\"\"\"\n",
         "\n",
+        "SQL_PROMPT = \"\"\"\n",
+        "Write code in SQL for extracting data based on the user's question\n",
+        "\"\"\""
+      ],
+      "metadata": {
+        "id": "B6dGxRmnyJHS"
+      },
+      "execution_count": null,
+      "outputs": [],
+      "id": "B6dGxRmnyJHS"
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "The following block is responsible for the logic of the agent and the output that it produces."
+      ],
+      "metadata": {
+        "id": "bWNgj913yo8h"
+      },
+      "id": "bWNgj913yo8h"
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "789b46d9-2189-4d3c-8f77-61b4675bf950"
+      },
+      "outputs": [],
+      "source": [
+        "# Replace this with your actual LLM setup\n",
+        "# Example:\n",
+        "# from langchain_openai import AzureChatOpenAI\n",
+        "# model = AzureChatOpenAI(...)\n",
+        "\n",
+        "def sql_code(files, question):\n",
+        "    try:\n",
+        "        dfs = [pd.read_csv(f.name) for f in files]\n",
+        "    except Exception as e:\n",
+        "        return f\"❌ Could not read CSVs: {e}\", \"\"\n",
+        "    try:\n",
+        "        agent = create_pandas_dataframe_agent(\n",
+        "        llm=model,\n",
+        "        df=df,\n",
+        "        verbose=True,\n",
+        "        agent_type=AgentType.ZERO_SHOT_REACT_DESCRIPTION,\n",
+        "        allow_dangerous_code=True,\n",
+        "        handle_parsing_errors=True,  # 👈 this is the fix\n",
+        "      )\n",
+        "\n",
+        "\n",
+        "\n",
+        "\n",
         "# --- Agent Logic ---\n",
         "def ask_agent(files, question):\n",
         "    try:\n",
@@ -141,8 +249,23 @@
         "        return output, trace\n",
         "\n",
         "    except Exception as e:\n",
-        "        return f\"❌ Agent error: {e}\", \"\"\n",
-        "\n",
+        "        return f\"❌ Agent error: {e}\", \"\"\n"
+      ],
+      "id": "789b46d9-2189-4d3c-8f77-61b4675bf950"
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "The next section creates the GUI and starts the programme."
+      ],
+      "metadata": {
+        "id": "-oxO_Rkq0ZMv"
+      },
+      "id": "-oxO_Rkq0ZMv"
+    },
+    {
+      "cell_type": "code",
+      "source": [
         "# --- Gradio UI ---\n",
         "with gr.Blocks(\n",
         "    css=\"\"\"\n",
@@ -193,11 +316,13 @@
         "\n",
         "    with gr.Row(equal_height=True):\n",
         "        file_input = gr.File(label=\"📁 Upload CSV(s)\", file_types=[\".csv\"], file_count=\"multiple\")\n",
-        "        question_input = gr.Textbox(\n",
-        "    label=\"💬 Ask Your Data\",\n",
-        "    placeholder=\"e.g., What is the trend for revenue over time?\",\n",
-        "    lines=9\n",
-        ")\n",
+        "        with gr.Column():\n",
+        "            question_input = gr.Textbox(\n",
+        "        label=\"💬 Ask Your Data\",\n",
+        "        placeholder=\"e.g., What is the trend for revenue over time?\",\n",
+        "        lines=2\n",
+        "    )\n",
+        "            sql_code = gr.Textbox(label=\"SQL code\",placeholder='SQL code will appear here',lines=2)\n",
         "\n",
         "\n",
         "    ask_button = gr.Button(\"💡 Analyze\")\n",
@@ -209,7 +334,49 @@
         "    )\n",
         "\n",
         "demo.launch(share=True)"
-      ]
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/",
+          "height": 591
+        },
+        "outputId": "e6ff2cac-90fc-44ea-bad5-20a66cccab56",
+        "id": "QXUpnFMdPlQA"
+      },
+      "execution_count": null,
+      "outputs": [
+        {
+          "output_type": "stream",
+          "name": "stdout",
+          "text": [
+            "Colab notebook detected. To show errors in colab notebook, set debug=True in launch()\n",
+            "* Running on public URL: https://1b7614b12579466c16.gradio.live\n",
+            "\n",
+            "This share link expires in 1 week. For free permanent hosting and GPU upgrades, run `gradio deploy` from the terminal in the working directory to deploy to Hugging Face Spaces (https://huggingface.co/spaces)\n"
+          ]
+        },
+        {
+          "output_type": "display_data",
+          "data": {
+            "text/plain": [
+              "<IPython.core.display.HTML object>"
+            ],
+            "text/html": [
+              "<div><iframe src=\"https://1b7614b12579466c16.gradio.live\" width=\"100%\" height=\"500\" allow=\"autoplay; camera; microphone; clipboard-read; clipboard-write;\" frameborder=\"0\" allowfullscreen></iframe></div>"
+            ]
+          },
+          "metadata": {}
+        },
+        {
+          "output_type": "execute_result",
+          "data": {
+            "text/plain": []
+          },
+          "metadata": {},
+          "execution_count": 13
+        }
+      ],
+      "id": "QXUpnFMdPlQA"
     },
     {
       "cell_type": "code",

README.md

@@ -58,7 +58,7 @@ Choose the **Data Upload** menu and choose a csv file from your computer or drag
 
 In the windows below choose an **Industries**, **Topics** and **Requirenments** for better results. You can add comments in a seperate window.
 
-<image src='Images/image.png' alt='uploading data' width=250>
+<image src='assets/image.png' alt='uploading data' width=250>
 
 **Receive the analysed data**
 
@@ -68,7 +68,7 @@ In the **Dashboard** tab you can see the the graphs of the graphs of some distri
 
 In the **Chat** tab you can ask the bot about the details of the data.
 
-<image src='Images/image-1.png' alt='uploading data' width=300>
+<image src='assets/image-1.png' alt='uploading data' width=300>
 
 
 ## <a name='architecture'></a>Architecture
@@ -82,13 +82,13 @@ After the analysis is completed the results are received in two tabs: **Data Bra
 1) General overview of the data is presented as well as the methodology of approaching the dataset
 
 <p align='center'>
-<image src='Images/executive_summary.png' alt='executive summary' width=500>
+<image src='assets/executive_summary.png' alt='executive summary' width=500>
 </p>
 
 2) Recommendations on possible aspects of the data are generated
 
 <p align='center'>
-<image src='Images/business_insights.png' alt='business insights' width=500>
+<image src='assets/business_insights.png' alt='business insights' width=500>
 </p>
 
 3) a conclusive overview of the data and statistical insights are presented
@@ -96,10 +96,10 @@ After the analysis is completed the results are received in two tabs: **Data Bra
 
 
 <p align='center'>
-<image src='Images/Methodology.png' alt='Methodology' width=500 />
-<image src='Images/Data_evidence.png' alt='Data evidence' width=500 />
-<image src='Images/Statistical_insights.png' alt='Statistical insights' width=500 />
-<image src='Images/categorical_distribution.png' alt='categorical distribution' width=500 />
+<image src='assets/Methodology.png' alt='Methodology' width=500 />
+<image src='assets/Data_evidence.png' alt='Data evidence' width=500 />
+<image src='assets/Statistical_insights.png' alt='Statistical insights' width=500 />
+<image src='assets/categorical_distribution.png' alt='categorical distribution' width=500 />
 </p>
 
 
@@ -108,21 +108,21 @@ After the analysis is completed the results are received in two tabs: **Data Bra
 Brief overview of the data with only the most important metrics and figures, such as:
  * file information
 <p align='center'>
-<image src='Images/file_information.png' alt='file_information' width=500 />
+<image src='assets/file_information.png' alt='file_information' width=500 />
 </p>
 
  * number of columns of each type (numerical, categorical and temporal)
 
  * data quality and statistical summary
 <p align='center'>
- <image src='Images/analysis_summary.png' alt='analysis_summary' width=500 />
+ <image src='assets/analysis_summary.png' alt='analysis_summary' width=500 />
 </p>
 
 Finally, graphs of the most important variables are presented.
 <p align='center'>
- <image src='Images/graph1.png' alt='graph1' width=225 />
- <image src='Images/graph2.png' alt='graph2' width=250 />
- <image src='Images/graph3.png' alt='graph3' width=225 />
+ <image src='assets/graph1.png' alt='graph1' width=225 />
+ <image src='assets/graph2.png' alt='graph2' width=250 />
+ <image src='assets/graph3.png' alt='graph3' width=225 />
 </p>
 
 ## <a name='requirenments--starting-procedures'></a>Requirenments & Starting Procedures