Spaces:
Sleeping
Sleeping
| title: Carbon Dioxide Removal Toolkit | |
| emoji: 🌍 | |
| colorFrom: green | |
| colorTo: blue | |
| sdk: streamlit | |
| sdk_version: "1.58.0" | |
| app_file: app.py | |
| pinned: false | |
| # Carbon Dioxide Removal Optimiser | |
| A Streamlit app for optimising carbon removal portfolios across different methods and constraints. | |
| --- | |
| ## Table of contents | |
| - [Data Maintenance Guide](#data-maintenance-guide) | |
| - [How it works](#how-it-works-overview) | |
| - [App tabs](#app-tabs) | |
| - [New resources](#new-resources) | |
| - [Scenarios (save and compare)](#scenarios-save-and-compare) | |
| - [File structure](#file-structure-what-each-data-file-does) | |
| - [Understanding inputs.csv](#understanding-inputscsv) | |
| - [Option 1: Hugging Face](#option-1-edit-directly-on-hugging-face-no-local-setup-needed) | |
| - [Option 2: Local](#option-2-work-locally) | |
| - [Quick reference](#quick-reference-where-is-each-parameter) | |
| - [Maintenance cases](#maintenance-cases) | |
| - [Before you start](#before-you-start) | |
| - [Case 1 - Modify existing coefficient values](#case-1--modify-existing-coefficient-values) | |
| - [Case 2 - Add an existing resource to an existing method](#case-2--add-an-existing-resource-to-an-existing-method) | |
| - [Case 3 - Add a new CDR method](#case-3--add-a-new-cdr-method) | |
| - [Case 4 - Add a new resource](#case-4--add-a-new-resource) | |
| - [Case 5 - Add a new resource group](#case-5--add-a-new-resource-group-new-category) | |
| - [Case 6 - Change whether a resource is optional](#case-6--change-whether-a-resource-is-optional-secondary-for-a-method) | |
| - [Case 7 - Rename a method or resource](#case-7--rename-a-method-or-resource) | |
| - [Case 8 - Edit tooltip, method note, and section intro texts](#case-8--edit-tooltip-method-note-and-section-intro-texts) | |
| - [Case 9 - Remove a method or resource](#case-9--remove-a-method-or-resource) | |
| - [Case 10 - Change the display order](#case-10--change-the-display-order-of-methods-or-resources) | |
| - [After any change - checklist](#after-any-change--checklist) | |
| - [Library Maintenance Guide](#library-maintenance-guide) | |
| - [What is a library?](#what-is-a-library-dependency) | |
| - [The key libraries](#the-key-libraries-and-what-they-do) | |
| - [Dependency chain](#dependency-chain-what-depends-on-what) | |
| - [When to update](#when-to-update) | |
| - [How to safely update](#how-to-safely-update-a-library) | |
| - [Install on a new machine](#how-to-install-all-libraries-on-a-new-machine) | |
| - [Check for outdated libraries](#checking-for-outdated-libraries) | |
| - [Check for vulnerabilities](#checking-for-known-security-vulnerabilities) | |
| - [Using AI to handle updates](#using-an-ai-assistant-to-handle-updates) | |
| - [Summary: update strategy](#summary-update-strategy) | |
| --- | |
| ## Data Maintenance Guide | |
| This guide is for Carbon Gap staff who need to update the tool's data: **no coding experience required** for most tasks. | |
| --- | |
| ### How it works (overview) | |
| All the data the tool uses (CDR methods, resources, coefficients) comes from a single file: | |
| ``` | |
| data/inputs.csv | |
| ``` | |
| Every time the app starts, it computes a fingerprint (hash) of the CSV and compares it to the last known value. If they differ, meaning the CSV has been modified, it automatically regenerates all the data files used by the application before loading. You do not need to run anything manually. | |
| **Automatic method deactivation:** if a CDR method requires resources that the user has not added in the Resource inputs tab, the method is automatically disabled in the Method constraints and Portfolio generation tabs. A "Why is this method disabled?" indicator is shown next to the method. This is expected behavior: no data change is needed. | |
| --- | |
| ### App tabs | |
| The app has seven tabs, accessible from the left sidebar: | |
| | Tab | Name | Purpose | | |
| |-----|------|---------| | |
| | Welcome | Welcome | Introduction, instructions, and a quick-start guide | | |
| | Resource inputs | Resource inputs | Set the available quantity of each resource; add new resource batches | | |
| | Method constraints | Method constraints | Activate or deactivate CDR methods; set caps and constraints per method | | |
| | Resource coefficients | Resource coefficients | View and adjust the coefficient values (min / median / max) used by the optimiser | | |
| | Portfolio generation | Portfolio generation | Run the LP optimiser and view results (charts, tables, resource usage) | | |
| | Sensitivity analysis | Sensitivity analysis | Analyse how binding each resource constraint is; identify bottlenecks | | |
| | Scenario comparison | Scenario comparison | Save the current scenario, load a previous one, and compare multiple runs side by side | | |
| --- | |
| ### New resources | |
| In the **Resource inputs** tab, users can add new resource batches: named resources that substitute for a standard or "Other" reference resource within a specific CDR method. | |
| For example, a user might add a batch called "Basalt A" linked to the reference group "Basalt mineral" and assign it to the Enhanced weathering method. The optimiser will then use the batch's new coefficients instead of the standard reference values for that method. | |
| New batches are stored in the session and saved when exporting a scenario. They do **not** require any change to `inputs.csv`. | |
| --- | |
| ### Scenarios (save and compare) | |
| In the **Portfolio generation** tab (Export & Save sub-tab), users can: | |
| - **Export** the current inputs only as an Excel `.xlsx` file, or the full scenario (inputs, constraints, coefficients, and optimisation results) as a `.json` file or a `.zip` of CSVs | |
| - **Import** a previously exported Excel file to reload inputs into all tabs, or a previously exported `.json`/`.zip` scenario file to restore a complete session and re-run the optimisation | |
| In the **Scenario comparison** tab, users can: | |
| - Upload one or more saved scenario files (`.json` or `.zip`) and **compare** them side by side in the charts | |
| Scenario files are standalone and portable. Exporting a `.json` or `.zip` preserves the full state including new resource batches; the Excel format covers only the inputs and can be opened directly in Excel. | |
| --- | |
| ### File structure (what each data file does) | |
| | File | What it is | Do you edit it? | | |
| |------|-----------|-----------------| | |
| | `data/inputs.csv` | **Main data file**: all methods, resources and coefficients | ✅ Yes | | |
| | `data/ui_text.xlsx` | Tooltip texts (on hover), "why is this method disabled?" notes, and each tab's intro paragraph | ✅ Yes | | |
| | `data/update.py` | Orchestration script: runs both generators below and updates the hash file | ❌ No | | |
| | `data/generate_model_data.py` | Generates `cost_sliders.json`, `method_groups.json`, `secondary_resources.json` from the CSV | ❌ No | | |
| | `data/generate_resource_template.py` | Generates `resource_template.xlsx` from the CSV | ❌ No | | |
| | `data/cost_sliders.json` | Auto-generated. Coefficients used by the app | ❌ No (auto) | | |
| | `data/method_groups.json` | Auto-generated. CDR method categories | ❌ No (auto) | | |
| | `data/secondary_resources.json` | Auto-generated. Optional resource assignments | ❌ No (auto) | | |
| | `data/resource_template.xlsx` | Auto-generated. Excel template for users | ❌ No (auto) | | |
| | `data/.csv_hash` | Auto-generated. Stores a fingerprint of `inputs.csv` to detect changes on restart | ❌ No (auto) | | |
| > ⚠️ **Never edit the auto-generated files directly**: your changes will be overwritten the next time `update.py` is run. | |
| ### Understanding `inputs.csv` | |
| Each row represents **one resource used by one CDR method**, with the following columns: | |
| | Column | Description | Example | | |
| |--------|-------------|---------| | |
| | `Implementation` | Name of the CDR method | `Afforestation/reforestation` | | |
| | `Method group` | Category the method belongs to | `Forestry` | | |
| | `Resource` | Name of the resource used | `Other land` | | |
| | `Resource group` | Category the resource belongs to | `Land` | | |
| | `Type` | Physical type of the resource | `Physical` | | |
| | `Unit` | Unit of the coefficient (per MtCO₂) | `Mha/MtCO2` | | |
| | `min` | Minimum coefficient value | `0.002` | | |
| | `median` | Default (median) coefficient value | `0.094927` | | |
| | `max` | Maximum coefficient value | `0.384615` | | |
| | `Is secondary` | Whether this resource is optional for this method | `FALSE` | | |
| > **Primary resources** (`Is secondary = FALSE`) are always active in the model. | |
| > **Secondary resources** (`Is secondary = TRUE`) are optional: users can toggle them on in the app. | |
| > **Naming convention: "Other" resources:** any resource whose name starts with `Other ` (capital O, followed by a space - e.g. `Other land`, `Other water`) is **hidden from the standard resource display** in Tabs 1 and 3. It still functions normally in the model and remains available as a reference group in the "Add resource" dialog. Use this convention intentionally when you want a resource to be inherited by new user-defined batches but not shown as a standalone input row. | |
| > **Naming convention: "Other" methods:** any CDR method whose name starts with `Other` receives special treatment in the optimiser. Instead of appearing as a fixed method, the LP automatically creates **one variant per new resource batch** the user adds that is linked to that method. This is intended for generic CDR pathways where the type of resource is user-defined. If you add a method using this naming convention, make sure it is intentional. | |
| > **Resource group and OR-substitutable resources:** when a CDR method has two or more **active** resources (primary, or secondary once toggled on) that belong to the same `Resource group`, the optimiser treats them as **interchangeable (OR)**. The method can satisfy its resource need from either one, or any mix of the two, rather than requiring all of them simultaneously. Users will see a hint like *"only one of Other land, Arable land is required"* in the "Why is this method disabled?" panel for primary siblings (secondary siblings are pooled by the optimiser the same way, but this specific hint currently only covers primary resources - see [Case 6](#case-6--change-whether-a-resource-is-optional-secondary-for-a-method)). This behaviour is automatic: it is triggered purely by two active resources sharing the same `Resource group` for the same method, whether or not they are primary. **If you add a resource to an existing group for a method that already has a resource active in that group, you are creating an OR relationship: make sure this is intentional.** If it is not, create a new resource group instead (see Case 5). | |
| --- | |
| There are two ways to update the data, depending on whether you are working directly on Hugging Face or on a local copy of the project: | |
| ### Option 1: Edit directly on Hugging Face (no local setup needed) | |
| This is the simplest option for non-coders with admin access to the Hugging Face Space. | |
| **Steps:** | |
| 1. Go to the Hugging Face **Space** (https://huggingface.co/CGdev: Carbon Dioxide Removal Toolkit) → click the **Files** tab | |
| 2. Open `data/inputs.csv` and click **Edit** (pencil icon) | |
| *OR* | |
| Upload a new version of the file: click **Delete** `data/inputs.csv`, then go to **data** folder → click **+ Contribute** → Upload Files | |
| 3. Commit the change directly on Hugging Face (checkbox *Commit directly to the main branch*): the Space will restart automatically | |
| 4. **Check the logs** to confirm the update ran: | |
| - In the Space, click: **Logs** (The button at the very top, next to *Running*) | |
| - You should see: | |
| ``` | |
| inputs.csv has changed - running data/update.py ... | |
| [... update details ...] | |
| Data files updated successfully. | |
| ``` | |
| 5. If the logs show this message, the app is running with the new data. If not, something went wrong: try Option 2 or contact a developer. You may also not have waited long enough or, conversely, you may have waited too long before checking the logs: check whether the app has successfully applied your changes. | |
| > **Note:** with this option, only `inputs.csv` needs to be uploaded. The app handles the rest automatically on restart. | |
| ### Option 2: Work locally | |
| Use this option when making bigger changes where you want to preview the result before publishing. | |
| **Step 1: Get the project on your machine** | |
| Before starting, make sure you have **Python** (version 3.9 or later) and **git** installed on your machine: | |
| - Python: https://www.python.org/downloads/ (tick *Add Python to PATH* during installation on Windows) | |
| - git - Mac: https://git-scm.com/download/mac - Windows: https://git-scm.com/download/win | |
| Then in a terminal, navigate to the folder where you want to put the project and run: | |
| ``` | |
| git clone <repository URL> | |
| ``` | |
| Example: | |
| ``` | |
| git clone https://huggingface.co/spaces/CarbonGap/CRRA_Optimization_Tool_V2 | |
| ``` | |
| **Step 1b: Set up the Python environment (first time only)** | |
| If you just cloned the repo, the venv/ folder doesn't exist yet. Create it by running in the terminal: | |
| ``` | |
| python -m venv venv | |
| ``` | |
| Then activate it and install the dependencies: | |
| ``` | |
| source venv/bin/activate # Mac / Linux | |
| venv\Scripts\activate # Windows | |
| ``` | |
| You should now see `(venv)` at the start of your terminal line. | |
| ``` | |
| pip install -r requirements.txt | |
| ``` | |
| You only need to do this once per machine. After that, just activate the environment (Step 3) each time. | |
| **Step 2: Edit the CSV** | |
| Open `data/inputs.csv` in Excel or any spreadsheet editor. Make your changes and save. | |
| > ⚠️ **Commas in names:** if a method name, resource name, or group name contains a comma, the CSV parser will misread it. Wrap the value in double quotes in the CSV (e.g. `"Biochar, pyrolysis"`) or, preferably, avoid commas entirely and use a dash or slash instead. | |
| **Step 3: Open a terminal in the project folder** | |
| You can use any of the following: | |
| - In your code editor (VS Code, PyCharm, etc.): open the integrated terminal, it's usually already pointing to the project folder | |
| - Mac: right-click the project folder → **New Terminal at Folder** | |
| - Windows: open the folder, click in the address bar, type `cmd`, press Enter | |
| Activate the Python environment: | |
| ``` | |
| source venv/bin/activate # Mac / Linux | |
| venv\Scripts\activate # Windows | |
| ``` | |
| **Step 4: Start the app to verify** | |
| ``` | |
| streamlit run app.py | |
| ``` | |
| The app detects that the CSV has changed and automatically runs the update script before loading. Check the terminal output: it should say `Data files updated successfully.` Then browse the app to confirm the change looks correct. | |
| > If the auto-update doesn't seem to work (no message in the terminal), you can trigger it manually as a fallback: `python data/update.py` | |
| **Step 5: Commit and push** | |
| In the terminal: | |
| ``` | |
| git add -A -- ':!venv/' | |
| git commit -m "Your description here (e.g. Update coefficients for Afforestation/reforestation)" | |
| git push | |
| ``` | |
| Hugging Face will detect the push and rebuild the Space automatically. | |
| --- | |
| ### Quick reference: where is each parameter? | |
| | I want to change… | File | Where exactly | | |
| |---|---|---| | |
| | A coefficient value (min / median / max) | `data/inputs.csv` | Columns `min`, `median`, `max`: find the row by `Implementation` + `Resource` | | |
| | The name of a CDR method | `data/inputs.csv` | Column `Implementation` | | |
| | Which group a CDR method belongs to | `data/inputs.csv` | Column `Method group` | | |
| | The name of a resource | `data/inputs.csv` | Column `Resource` | | |
| | Which group a resource belongs to | `data/inputs.csv` | Column `Resource group` | | |
| | The unit of a resource group | `data/inputs.csv` | Column `Unit`: must be consistent across all rows for that group | | |
| | Whether a resource is optional for a method | `data/inputs.csv` | Column `Is secondary`: `TRUE` = optional, `FALSE` = always active | | |
| | The display order of methods or resources | `data/inputs.csv` | Row order: in the app, groups are displayed in the order their first row is encountered in the CSV; within a group, methods or resources are displayed in the order their rows appear. Move rows up/down to reorder | | |
| | The app's UI colours (background, buttons, links) | `.streamlit/config.toml` | `primaryColor`, `backgroundColor`, `secondaryBackgroundColor`, `textColor` | | |
| | The chart colours (Plotly graphs) | `config/config.py` | `CRRA_COLORS` dictionary | | |
| | The app's UI font | `.streamlit/config.toml` | `font` - accepted values: `"sans-serif"`, `"serif"`, `"monospace"` | | |
| | The chart font (Plotly graphs) | `config/config.py` | `CRRA_FONT` variable | | |
| | The app's UI colours and font (fine-tuned overrides) | `assets/styles.css` | `:root` block at the top of the file - change the hex values next to each variable name | | |
| | The tooltip text on a method name | `data/ui_text.xlsx` | Sheet **Methods**: column `Tooltip` | | |
| | The tooltip text on a resource name | `data/ui_text.xlsx` | Sheet **Resources**: column `Tooltip` | | |
| | The text shown in "Why is this method disabled?" for a specific method | `data/ui_text.xlsx` | Sheet **Methods**: column `Note`, supports basic HTML (see [Case 8](#case-8--edit-tooltip-method-note-and-section-intro-texts)) | | |
| | The "OBJECTIVE" intro paragraph at the top of a tab | `data/ui_text.xlsx` | Sheet **Section Intros**: column `Text`, row selected by `Tab`, supports basic HTML (see [Case 8](#case-8--edit-tooltip-method-note-and-section-intro-texts)) | | |
| | UI spacing, font sizes, button styles | `assets/styles.css` | Standard CSS: search for the element name | | |
| | Python library versions | `requirements.txt` | One line per library: see Library Maintenance Guide below | | |
| > After any change to `inputs.csv`, start the app - it detects the change and updates automatically. After changes to `config/config.py`, `assets/styles.css`, or `data/ui_text.xlsx`, simply restart the app. | |
| --- | |
| ## Maintenance cases | |
| ### Before you start | |
| Each case below covers two workflows: | |
| - **Option 1 - Hugging Face:** edit directly in the browser, no local setup needed. | |
| - **Option 2 - Local:** make changes on your machine, then push to Hugging Face. | |
| For Option 2, make sure the project is cloned and the Python environment is set up (see [Option 2](#option-2-work-locally) at the top of this guide). Before running any command in the terminal, activate the environment: | |
| ``` | |
| source venv/bin/activate # Mac / Linux | |
| venv\Scripts\activate # Windows | |
| ``` | |
| You should see `(venv)` at the start of your terminal line. | |
| > **For all Option 2 cases below:** after editing the CSV, start the app (`streamlit run app.py`) to verify - it detects the change and runs the update automatically. If the auto-update doesn't trigger (no message in the terminal), run `python data/update.py` manually as a fallback. | |
| --- | |
| ### Case 1 - Modify existing coefficient values | |
| **When to use:** The literature has been updated and you want to change the min/median/max values for an existing method-resource pair. | |
| **Option 1 - On Hugging Face** | |
| 1. In the Space, click the **Files** tab. | |
| 2. Open `data/inputs.csv` → click the **pencil icon** to edit inline. | |
| 3. Find the row with the correct `Implementation` and `Resource`. Edit the `min`, `median`, and/or `max` columns. | |
| 4. Scroll down → **Commit directly to main branch** → **Commit changes to main**. | |
| 5. The Space restarts. Click **Logs** to confirm: `Data files updated successfully.` | |
| **Option 2 - Locally** | |
| 1. Open `data/inputs.csv` in Excel or a text editor. | |
| 2. Find the row with the correct `Implementation` and `Resource`. | |
| 3. Edit the `min`, `median`, and/or `max` columns. | |
| 4. Save the file, then start the app to verify: | |
| ``` | |
| streamlit run app.py | |
| ``` | |
| 5. Commit and push: | |
| ``` | |
| git add -A -- ':!venv/' | |
| git commit -m "Update coefficients for [method name] and [resource name]" | |
| git push | |
| ``` | |
| ### Case 2 - Add an existing resource to an existing method | |
| **When to use:** A CDR method should now also use a resource that is already tracked for other methods (e.g., water is already listed for other methods, and you want to add it to Afforestation/reforestation too). If the resource doesn't exist at all yet, use Case 4 first. | |
| > ⚠️ **This may create an OR relationship:** if the method already has another active resource (primary, or secondary and toggled on) in the same `Resource group` as the one you're adding, the optimiser will treat the two as interchangeable (the method can run on either one, not both) as soon as both are active - marking the new row `Is secondary = TRUE` only delays this until a user enables it, it does not prevent it. See [Understanding inputs.csv](#understanding-inputscsv) for details. If this is not the intended behavior, use a different `Resource group` for the new row (Case 5). | |
| **Option 1 - On Hugging Face** | |
| 1. In the Space, click **Files** → open `data/inputs.csv` → click the pencil icon. | |
| 2. Add a new row, making sure to include all the expected columns (see [Understanding inputs.csv](#understanding-inputscsv)). | |
| 3. Commit directly to main → Space restarts. Check Logs for `Data files updated successfully.` | |
| **Option 2 - Locally** | |
| 1. Open `data/inputs.csv` and add a new row (see [Understanding inputs.csv](#understanding-inputscsv)). | |
| 2. Save the file, then start the app to verify: | |
| ``` | |
| streamlit run app.py | |
| ``` | |
| 3. Commit and push: | |
| ``` | |
| git add -A -- ':!venv/' | |
| git commit -m "Add [resource name] to [method name]" | |
| git push | |
| ``` | |
| ### Case 3 - Add a new CDR method | |
| **When to use:** A new carbon removal approach needs to be added to the tool. | |
| > ⚠️ **"Other" method naming:** if the method name starts with `Other` (e.g. `Other CDR`), the optimiser treats it specially: it creates one variant per new resource batch that the user links to this method, instead of running it as a fixed method. Only use this naming if that expansion behavior is intentional. For a standard CDR method with defined resource inputs, use a name that does not start with `Other`. | |
| **Option 1 - On Hugging Face** | |
| 1. In the Space, click **Files** → open `data/inputs.csv` → click the pencil icon. | |
| 2. Add **one row per resource** the new method uses (see [Understanding inputs.csv](#understanding-inputscsv)). Use an existing `Method group` or create a new one. | |
| 3. Place the rows where you want the method to appear: display order follows row order in the CSV. | |
| 4. Commit directly to main → Space restarts. Check Logs for `Data files updated successfully.` | |
| **Option 2 - Locally** | |
| 1. Open `data/inputs.csv`. | |
| 2. Add one row per resource used by the new method. Fill in all columns (see [Understanding inputs.csv](#understanding-inputscsv)). | |
| 3. Save the file, then start the app to verify: | |
| ``` | |
| streamlit run app.py | |
| ``` | |
| 4. Commit and push: | |
| ``` | |
| git add -A -- ':!venv/' | |
| git commit -m "Add new CDR method: [method name]" | |
| git push | |
| ``` | |
| --- | |
| ### Case 4 - Add a new resource | |
| **When to use:** A resource that doesn't exist yet needs to be used (e.g., a new type of mineral or energy). | |
| > ⚠️ **"Other" resource naming:** if you name the resource starting with `Other ` (capital O + space - e.g. `Other land`, `Other water`), it will be **hidden from the standard resource display** in the app. Users won't see it as a standalone input row, but it will still be available as a reference group when they use the "Add resource" dialog. Only use this prefix if you specifically want the resource to be hidden from direct input. | |
| > ⚠️ **Adding to an existing Resource group may create an OR relationship:** if the method you are adding this resource to already has another active resource (primary, or secondary and toggled on) in the same `Resource group`, the optimiser will treat the two resources as interchangeable as soon as both are active. See [Understanding inputs.csv](#understanding-inputscsv) for details. If this is not the intended behavior, create a new Resource group instead (Case 5). | |
| **Option 1 - On Hugging Face** | |
| 1. Decide which **Resource group** the new resource belongs to. If it belongs to an existing group, use the exact same name. If it needs its own group, see Case 5. | |
| 2. In the Space, click **Files** → open `data/inputs.csv` → pencil icon. | |
| 3. Add one row per method that uses this resource (see column descriptions in Case 2). | |
| 4. **Check the unit:** all resources in a group must share the same unit (e.g., all `Land` resources use `Mha/MtCO2`). If the unit differs, create a new group (Case 5). | |
| 5. Commit directly to main → Space restarts. Check Logs. | |
| **Option 2 - Locally** | |
| 1. Decide which Resource group the new resource belongs to. | |
| 2. Open `data/inputs.csv` and add one row per method that uses this resource. | |
| 3. Verify the unit is consistent with the rest of the group. | |
| 4. Save, then start the app to verify: | |
| ``` | |
| streamlit run app.py | |
| ``` | |
| 5. Commit and push: | |
| ``` | |
| git add -A -- ':!venv/' | |
| git commit -m "Add new resource: [resource name]" | |
| git push | |
| ``` | |
| --- | |
| ### Case 5 - Add a new resource group (new category) | |
| **When to use:** Your new resource doesn't fit any existing category. Also use this case when the new resource belongs to the same physical category as an existing one but should **not** be treated as interchangeable with it by the optimiser: keeping it in a separate group avoids the automatic OR relationship (see [Understanding inputs.csv](#understanding-inputscsv)). | |
| **Option 1 - On Hugging Face** | |
| Same steps as Case 4, but write a **new group name** (never used before) in the `Resource group` column. The app will create a new section for it automatically. Place the rows where you want the group to appear: it appears at the position of its first row in the CSV. | |
| **Option 2 - Locally** | |
| Same steps as Case 4, but write a **new group name** in the `Resource group` column. The new group appears in the app at the position of its first row in the CSV. | |
| --- | |
| ### Case 6 - Change whether a resource is optional (secondary) for a method | |
| **When to use:** You want to make a resource optional (user can enable/disable it) or make it always required. | |
| > ⚠️ **This may create an OR relationship:** the optimiser pools any two resources that are both active (primary, or secondary and toggled on) in the same `Resource group` for the same method - regardless of which way you're changing `Is secondary`. So toggling a secondary resource on (in the app) can create an OR relationship with a sibling in the same group just as much as setting `Is secondary = FALSE` in the CSV can. See [Understanding inputs.csv](#understanding-inputscsv) for details. If this is not intended, consider moving the resource to a different group (Case 5). | |
| **Option 1 - On Hugging Face** | |
| 1. In the Space, click **Files** → open `data/inputs.csv` → pencil icon. | |
| 2. Find the row with the correct `Implementation` and `Resource`. | |
| 3. Change `Is secondary` to `TRUE` (optional) or `FALSE` (always active). | |
| 4. Commit directly to main → Space restarts. Check Logs. | |
| **Option 2 - Locally** | |
| 1. Open `data/inputs.csv`. | |
| 2. Find the row with the correct `Implementation` and `Resource`. | |
| 3. Change `Is secondary` to `TRUE` (optional) or `FALSE` (always active). | |
| 4. Save, then start the app to verify: | |
| ``` | |
| streamlit run app.py | |
| ``` | |
| 5. Commit and push: | |
| ``` | |
| git add -A -- ':!venv/' | |
| git commit -m "Update secondary status for [resource] in [method]" | |
| git push | |
| ``` | |
| --- | |
| ### Case 7 - Rename a method or resource | |
| **When to use:** Correcting a spelling mistake or updating nomenclature. | |
| > ⚠️ If users have saved scenario files (`.xlsx` or `.zip`) that reference the old name, those files will no longer load correctly after the rename. | |
| **Option 1 - On Hugging Face** | |
| Editing inline is not recommended for a rename because Find & Replace is not available in the browser editor. Instead: | |
| 1. Download `data/inputs.csv` from the Files tab. | |
| 2. Open it in Excel. Use **Find & Replace** (Ctrl+H / Cmd+H) to replace the old name with the new name in the `Implementation` / `Resource` columns and, if the group name changed, in `Method group` / `Resource group`. | |
| 3. Also download `data/ui_text.xlsx`, update the name in the `Method` or `Resource` column of the relevant sheet, and save. | |
| 4. Upload both files via **Files** → `data/` folder → **+ Contribute** → **Upload Files** → Commit. | |
| 5. Space restarts → check Logs. | |
| **Option 2 - Locally** | |
| 1. Open `data/inputs.csv`. Use **Find & Replace** (Ctrl+H / Cmd+H) to replace the old name everywhere in the file. | |
| 2. Open `data/ui_text.xlsx` and update the name in the `Method` or `Resource` column of the relevant sheet. Save. | |
| 3. Start the app to verify: | |
| ``` | |
| streamlit run app.py | |
| ``` | |
| 4. Commit and push: | |
| ``` | |
| git add -A -- ':!venv/' | |
| git commit -m "Rename [old name] to [new name]" | |
| git push | |
| ``` | |
| --- | |
| ### Case 8 - Edit tooltip, method note, and section intro texts | |
| **When to use:** You want to add, update, or remove any of the editable text in the app: hover tooltips, the guidance note shown under "Why is this method disabled?" for a specific method, or the "OBJECTIVE" intro paragraph at the top of a tab. | |
| Everything is in the same file, `data/ui_text.xlsx`, across three sheets: | |
| | Sheet | Columns | Controls | | |
| |---|---|---| | |
| | **Resources** | `Resource`, `Tooltip` | Hover tooltip on a resource name | | |
| | **Methods** | `Method`, `Tooltip`, `Note` | `Tooltip` = hover tooltip on a method name. `Note` = the message shown at the top of that method's "Why is this method disabled?" panel (Method constraints tab), e.g. explaining that Mineral OAE needs a specific mineral feedstock added as a new resource. Leave empty for no note. | | |
| | **Section Intros** | `Tab`, `Text` | The "OBJECTIVE" paragraph shown at the top of a tab. One row per tab: see the `Tab` key table below. | | |
| > ℹ️ **The "Why is this method disabled?" panel shows two types of text that come from different sources.** The resource hints, e.g. *"Other land: Add as new resource in Resource inputs"* or *"only one of Other land, Arable land is required"*, are **auto-generated** from `inputs.csv` (based on which resources are missing and whether an OR relationship exists). They cannot be changed in `ui_text.xlsx`. The optional free-text `Note` from `ui_text.xlsx` appears above those hints with a lightbulb icon: that is the only part you can edit here. To change the resource hints, you need to change the resource or group structure in `inputs.csv`. | |
| > ⚠️ **Exact name match:** the name in column `Resource` or `Method` must match **exactly** the name used in `inputs.csv` (same spelling, same capitalisation). If you rename a method or resource in the CSV, remember to update the name in `ui_text.xlsx` too. | |
| > ⚠️ **`Tab` keys, not tab titles:** the `Tab` column in **Section Intros** uses the internal page name, not the label shown in the sidebar. Use exactly one of: | |
| | `Tab` value | Corresponding tab in the app | | |
| |---|---| | |
| | `tab1_inputs` | Resource inputs | | |
| | `tab2_constraints` | Method constraints | | |
| | `tab3_coefficients` | Resource coefficients | | |
| | `tab4_optimisation` | Portfolio generation | | |
| | `tab5_sensitivity` | Sensitivity analysis | | |
| > A row with any other value is simply ignored. | |
| > ⚠️ **These texts contain HTML, not just plain text.** `Tooltip` cells are plain text, but `Note` and `Text` (Section Intros) are rendered as HTML, so you'll see markup like `<strong>bold</strong>`, `<br>` for a line break, and `<li>...</li>` for a bullet-style line. Keep these tags when editing around them: deleting a `<` or `>` by accident will make the raw tag appear as text on the page instead of being applied as formatting. You can add more of the same tags (`<strong>`, `<br>`, `<li>`) but avoid pasting anything more advanced (scripts, styles, other tags) since it is not sanitised before display. | |
| > ⚠️ **`tab2_constraints`'s text has a placeholder:** its `Text` cell contains `{CO2_UNIT}`, which the app replaces with the actual CO₂ unit (e.g. `MtCO₂`) when displayed. Keep the `{CO2_UNIT}` placeholder (braces included) if you want that value to appear; removing it just removes the unit from the sentence, it won't break anything. | |
| > If a cell is left empty, or a sheet/row is missing entirely, that piece of text is simply skipped (no tooltip / no note / no intro paragraph): nothing crashes. | |
| **Option 1 - On Hugging Face** | |
| The `.xlsx` file cannot be edited inline on Hugging Face. Instead: | |
| 1. In the Files tab, open the `data/` folder → download `ui_text.xlsx`. | |
| 2. Open it in Excel. Go to the relevant sheet (**Resources**, **Methods**, or **Section Intros**). | |
| 3. Find the row with the name (or `Tab` key) you want to edit. Fill in or change the `Tooltip` / `Note` / `Text` column. Leave it empty to remove that text. | |
| 4. Save the file. | |
| 5. Back in the Files tab, go to `data/` → **+ Contribute** → **Upload Files**. | |
| 6. Upload your edited `ui_text.xlsx` → Commit directly to main. | |
| 7. The Space restarts. **No update script runs for this file** - it is read directly by the app on each restart. | |
| **Option 2 - Locally** | |
| 1. Open `data/ui_text.xlsx` in Excel. | |
| 2. Go to the relevant sheet (**Resources**, **Methods**, or **Section Intros**). | |
| 3. Find the row with the name (or `Tab` key) you want to edit. Fill in or change the `Tooltip` / `Note` / `Text` column. Leave it empty to remove that text. | |
| 4. Save the file. | |
| 5. Restart the app (`Ctrl+C` then `streamlit run app.py`). **No need to run `update.py`.** | |
| 6. Commit and push: | |
| ``` | |
| git add data/ui_text.xlsx | |
| git commit -m "Update tooltip / method note / section intro texts" | |
| git push | |
| ``` | |
| --- | |
| ### Case 9 - Remove a method or resource | |
| **When to use:** A CDR method or resource is no longer relevant and should be removed from the tool entirely. | |
| > ⚠️ If users have saved scenario files (`.xlsx` or `.zip`) that reference the removed method or resource, those fields will be silently ignored on load: no error is raised, but the data will be missing from the restored session. | |
| **Option 1 - On Hugging Face** | |
| 1. Download `data/inputs.csv` from the Files tab. | |
| 2. Open it in Excel. Delete all rows where the `Implementation` column matches the method to remove, **or** all rows where the `Resource` column matches the resource to remove. | |
| - To remove a method: delete every row for that `Implementation` value. | |
| - To remove a resource from one method only: delete only the row(s) matching both the `Implementation` and `Resource`. | |
| - To remove a resource entirely: delete every row where the `Resource` column matches, across all methods. | |
| 3. Also open `data/ui_text.xlsx` and delete the corresponding row in the **Methods** or **Resources** sheet if a tooltip exists for the removed item. | |
| 4. Upload both modified files via **Files** → `data/` → **+ Contribute** → **Upload Files** → Commit directly to main. | |
| 5. Space restarts → check Logs for `Data files updated successfully.` | |
| **Option 2 - Locally** | |
| 1. Open `data/inputs.csv` and delete the relevant rows (see Option 1 above for which rows to target). | |
| 2. Open `data/ui_text.xlsx` and delete the corresponding tooltip row if one exists. Save. | |
| 3. Start the app to verify: | |
| ``` | |
| streamlit run app.py | |
| ``` | |
| 4. Commit and push: | |
| ``` | |
| git add -A -- ':!venv/' | |
| git commit -m "Remove [method name / resource name]" | |
| git push | |
| ``` | |
| --- | |
| ### Case 10 - Change the display order of methods or resources | |
| **When to use:** You want a method or resource group to appear earlier or later in the app (Resource inputs, Method constraints tabs...). | |
| **Option 1 - On Hugging Face** | |
| Moving rows is easier in Excel than in a browser editor: | |
| 1. Download `data/inputs.csv` from the Files tab. | |
| 2. Open it in Excel. Move the relevant rows up or down - the app reads the CSV from top to bottom. | |
| 3. Upload the updated file via **Files** → `data/` → **+ Contribute** → **Upload Files** → Commit. | |
| 4. Space restarts → check Logs for `Data files updated successfully.` | |
| **Option 2 - Locally** | |
| 1. Open `data/inputs.csv`. | |
| 2. Move the relevant rows up or down. | |
| 3. Save, then start the app to verify: | |
| ``` | |
| streamlit run app.py | |
| ``` | |
| 4. Commit and push: | |
| ``` | |
| git add -A -- ':!venv/' | |
| git commit -m "Reorder [methods/resources]" | |
| git push | |
| ``` | |
| --- | |
| ### After any change - checklist | |
| **For changes to `inputs.csv` (Cases 1–7, 9–10):** | |
| Option 1 (Hugging Face direct): | |
| - [ ] Edited and committed `inputs.csv` on Hugging Face | |
| - [ ] Checked the Space logs: confirmed `Data files updated successfully.` | |
| - [ ] Verified the change appears correctly in the app | |
| Option 2 (local): | |
| - [ ] Saved `inputs.csv` | |
| - [ ] Started the app locally (`streamlit run app.py`): confirmed `Data files updated successfully.` in the terminal | |
| - [ ] Verified the change appears correctly in the app | |
| - [ ] Committed all modified files (CSV, JSON files, templates) and pushed | |
| **For changes to `ui_text.xlsx` (Case 8):** | |
| Option 1 (Hugging Face direct): | |
| - [ ] Downloaded, edited, and re-uploaded `ui_text.xlsx` on Hugging Face | |
| - [ ] Space has restarted (no update script runs for this file: `Data files updated successfully.` will **not** appear in the logs; this is normal) | |
| - [ ] Verified the updated text appears correctly in the app | |
| Option 2 (local): | |
| - [ ] Saved `data/ui_text.xlsx` | |
| - [ ] Restarted the app (`Ctrl+C` then `streamlit run app.py`): no need to run `update.py` | |
| - [ ] Verified the updated text appears correctly in the app | |
| - [ ] Committed `data/ui_text.xlsx` and pushed | |
| --- | |
| ## Library Maintenance Guide | |
| This section explains how to keep the tool's software dependencies up to date, safely and without breaking anything. | |
| --- | |
| ### What is a library (dependency)? | |
| The tool is built on top of external software packages ("libraries") that handle things like displaying the interface, reading Excel files, or running the optimisation. These libraries are listed in `requirements.txt` with their exact version numbers (e.g., `streamlit==1.45.1`). | |
| Libraries sometimes release updates - fixing bugs, patching security holes, or adding new features. The risk is that an update can also change how a library behaves and break the tool. | |
| --- | |
| ### The key libraries and what they do | |
| These are the libraries that matter most for this tool: | |
| | Library | Role | Risk if updated | | |
| |---------|------|----------------| | |
| | **streamlit** | Powers the entire web interface (tabs, sliders, buttons, charts) - currently 1.58.0 | 🔴 High - UI layout and widget behavior can change | | |
| | **streamlit-option-menu** | The navigation sidebar menu | 🟡 Medium - must stay compatible with streamlit version | | |
| | **pandas** | Reads and processes CSV and Excel files | 🟡 Medium - occasional API changes | | |
| | **numpy** | Numerical operations (arrays, math) | 🟡 Medium - used by scipy and pandas | | |
| | **scipy** | Runs the LP optimisation (`linprog`) | 🟡 Medium - optimisation results could change | | |
| | **plotly** | All charts and interactive graphs | 🟡 Medium - chart API evolves | | |
| | **openpyxl** | Reads and writes Excel files (.xlsx) | 🟢 Low - stable API | | |
| | **pillow** | Image handling (indirect use) | 🟢 Low | | |
| | **requests / urllib3 / certifi** | Network and security | 🟢 Low - update freely for security patches | | |
| --- | |
| ### Dependency chain (what depends on what) | |
| Some libraries require specific versions of others to work. Updating one can force updates to others: | |
| ``` | |
| streamlit ──────────────► tornado (web server) | |
| ├───────────► altair (charts) | |
| ├───────────► pyarrow (data performance) | |
| ├───────────► narwhals (data compatibility) | |
| ├───────────► protobuf | |
| └───────────► cachetools, blinker, click, toml | |
| pandas ─────────────────► numpy | |
| ├───────────────► pyarrow | |
| └───────────────► python-dateutil, pytz, tzdata | |
| scipy ──────────────────► numpy | |
| plotly ─────────────────► narwhals | |
| openpyxl ──────────────► pillow | |
| ``` | |
| **Practical implication:** if you update `streamlit`, you may also need to update `tornado`, `altair`, `pyarrow`, etc. If you update `pandas`, you may need to update `numpy`. These cascading updates are handled automatically by pip (see below), but they can introduce unexpected changes. | |
| --- | |
| ### When to update | |
| #### ✅ Update immediately | |
| - **`pip-audit` flags a vulnerability** - run `pip-audit` regularly (see [Check for vulnerabilities](#checking-for-known-security-vulnerabilities)); any flagged library should be patched without delay. | |
| - **Security vulnerabilities** in `requests`, `urllib3`, `certifi`, or `pillow` - these are reported publicly and should be patched fast even before `pip-audit` catches them. | |
| - Check https://pypi.org/project/[library-name] or subscribe to GitHub security alerts. | |
| #### ✅ Update when needed | |
| - **Streamlit** - when a new feature in the app requires it, or when a critical bug in the current version is causing problems. | |
| - **pandas / numpy / scipy** - when a bug in the current version is causing incorrect data processing or optimisation results. | |
| - **openpyxl** - when Excel import/export breaks with newer `.xlsx` files. | |
| #### ❌ Do NOT update just because a newer version exists | |
| - "Latest is always better" is not true for production tools. An update that works fine in isolation can break interactions between libraries. | |
| - The tool has been tested with the versions in `requirements.txt`. Changing them without testing is risky. | |
| --- | |
| ### Checking for outdated libraries | |
| To see which installed libraries have newer versions available: | |
| ``` | |
| pip list --outdated | |
| ``` | |
| This shows a list with current and latest versions. Use this to **identify** what could be updated but don't update everything at once. | |
| --- | |
| ### How to safely update a library | |
| **Never update all libraries at once.** Update one at a time, test the tool after each one. | |
| **Step 1 - Back up the current working state** | |
| ``` | |
| cp requirements.txt requirements.txt.backup | |
| ``` | |
| **Step 2 - Simulate first (dry run)** | |
| Before applying anything, check whether the update would cause conflicts: | |
| ``` | |
| pip install --upgrade streamlit --dry-run | |
| ``` | |
| If pip reports conflicts, investigate before proceeding. | |
| **Step 3 - Update the library** | |
| ``` | |
| pip install --upgrade streamlit | |
| ``` | |
| Or to a specific version: | |
| ``` | |
| pip install streamlit==1.58.0 | |
| ``` | |
| **Step 4 - Verify no dependencies are broken** | |
| ``` | |
| pip check | |
| ``` | |
| This confirms that all installed packages are mutually compatible. If it reports any conflict, do not proceed to the next step - restore the backup instead (see Step 6). | |
| **Step 5 - Freeze the new versions to requirements.txt** | |
| ``` | |
| pip freeze > requirements.txt | |
| ``` | |
| `pip freeze` lists every library currently installed in the environment with its exact version number, and `> requirements.txt` writes that list into the file. This ensures that anyone who installs from `requirements.txt` later, including Hugging Face when the Space rebuilds, gets exactly the same versions you just tested. | |
| **Step 6 - Test the tool** | |
| - Start the app: `streamlit run app.py` | |
| - Go through every tab | |
| - Import and export an Excel file | |
| - Run an optimisation | |
| - Check that charts display correctly | |
| **Step 7 - If something breaks** | |
| Restore the backup: | |
| ``` | |
| pip install -r requirements.txt.backup | |
| cp requirements.txt.backup requirements.txt | |
| ``` | |
| **Step 8 - If everything works** | |
| Delete the backup file, it's no longer needed: | |
| ``` | |
| rm requirements.txt.backup | |
| ``` | |
| Then commit the updated `requirements.txt` and push. | |
| --- | |
| ### How to install all libraries on a new machine | |
| ``` | |
| python -m venv venv | |
| source venv/bin/activate # Mac/Linux | |
| # or: venv\Scripts\activate # Windows | |
| pip install -r requirements.txt | |
| ``` | |
| --- | |
| ### Checking for known security vulnerabilities | |
| Install the security scanner: | |
| ``` | |
| pip install pip-audit | |
| ``` | |
| Run it: | |
| ``` | |
| pip-audit | |
| ``` | |
| This will flag any library with a known security issue and suggest a fix. These should be addressed promptly. | |
| > **Recommended frequency:** run `pip-audit` at least once a year, even if the tool appears stable. Security vulnerabilities are discovered independently of whether the library has changed - a version that was safe last year may be flagged today. | |
| --- | |
| ### Using an AI assistant to handle updates | |
| Library updates sometimes require code changes alongside the pip update - for example, a deprecated function that was renamed, or a new argument that became required. Doing this manually requires reading changelogs, finding every affected line, and testing the result. | |
| An AI coding assistant such as **Claude Code** can handle this end-to-end: | |
| - Run `pip-audit` or `pip list --outdated` and paste the output into Claude Code | |
| - Describe the update you want to make (e.g., "update pandas and fix any deprecation warnings") | |
| - Claude Code will update the library, search the codebase for affected code, apply the necessary fixes, and update `requirements.txt` | |
| This is especially useful when: | |
| - A deprecation warning points to a function that is used in multiple places across the codebase | |
| - A major version bump changes how a library behaves (e.g., pandas 3.x changing its default string backend) | |
| - You are not comfortable navigating the code yourself | |
| > The AI handles the mechanical parts; you remain responsible for testing the result with `streamlit run app.py` before committing and pushing. | |
| --- | |
| ### Summary: update strategy | |
| ``` | |
| Security fix needed? ──► Update immediately (requests, urllib3, certifi, pillow) | |
| Something is broken? ──► Update only the relevant library, test thoroughly | |
| Nothing is broken? ──► Do not update | |
| ``` |