CRRA_Optimization_Tool_V2

Sleeping

App Files Files Community

CRRA_Optimization_Tool_V2 / README.md

CarlyneB

[FIX] No more derived method

7c445a7 about 1 month ago

31.4 kB

title: Carbon Dioxide Removal Toolkit
emoji: 🌍
colorFrom: green
colorTo: blue
sdk: streamlit
sdk_version: 1.58.0
app_file: app.py
pinned: false

Carbon Dioxide Removal Optimizer

A Streamlit app for optimizing carbon removal portfolios across different methods and constraints.

Data Maintenance Guide
Maintenance cases
Library Maintenance Guide

Data Maintenance Guide

This guide is for Carbon Gap staff who need to update the tool's data: no coding experience required for most tasks.

How it works (overview)

All the data the tool uses (CDR methods, resources, coefficients) comes from a single file:

data/inputs_aggregated.csv

Every time the app starts, it computes a fingerprint (hash) of the CSV and compares it to the last known value. If they differ, meaning the CSV has been modified, it automatically regenerates all the data files used by the application before loading. You do not need to run anything manually.

There are two ways to update the data, depending on whether you are working directly on Hugging Face or on a local copy of the project:

Option 1: Edit directly on Hugging Face (no local setup needed)

This is the simplest option for non-coders with admin access to the Hugging Face Space.

Steps:

Go to the Hugging Face Space (Carbon Dioxide Removal Toolkit) → click the Files tab
Open data/inputs_aggregated.csv and click Edit (pencil icon) OR Upload a new version of the file: click Delete data/inputs_aggregated.csv, then go to data folder → click + Contribute → Upload Files
Commit the change directly on Hugging Face (checkbox Commit directly to the main branch): the Space will restart automatically
Check the logs to confirm the update ran:
- In the Space, click: Logs (The button at the very top, next to Running)
- You should see:
```
inputs_aggregated.csv has changed - running data/update.py ...
[... update details ...]
Data files updated successfully.
```
If the logs show this message, the app is running with the new data. If not, something went wrong: try option 2 or contact a developer. You may also not have waited long enough or, conversely, you may have waited too long before checking the logs: check whether the app has successfully applied your changes.

Note: with this option, only inputs_aggregated.csv needs to be uploaded. The app handles the rest automatically on restart.

Option 2: Work locally

Use this option when making bigger changes where you want to preview the result before publishing.

Step 1: Get the project on your machine

If you do not already have git installed: - Mac: install it via https://git-scm.com/download/mac - Windows: install it via https://git-scm.com/download/win

Then in a terminal, navigate to the folder where you want to put the project and run:

git clone <repository URL>

Example:

git clone https://huggingface.co/spaces/CarbonGap/CRRA_Optimization_Tool

Step 1b: Set up the Python environment (first time only)

If you just cloned the repo, the venv/ folder doesn't exist yet. Create it by running in the terminal:

python -m venv venv

Then activate it and install the dependencies:

source venv/bin/activate    # Mac / Linux
venv\Scripts\activate       # Windows

You should now see (venv) at the start of your terminal line.

pip install -r requirements.txt

You only need to do this once per machine. After that, just activate the environment (Step 3) each time.

Step 2: Edit the CSV

Open data/inputs_aggregated.csv in Excel or any spreadsheet editor. Make your changes and save.

⚠️ Commas in names: if a method name, resource name, or group name contains a comma, the CSV parser will misread it. Wrap the value in double quotes in the CSV (e.g. "Biochar, pyrolysis") or, preferably, avoid commas entirely and use a dash or slash instead.

Step 3: Open a terminal in the project folder

You can use any of the following:

In your code editor (VS Code, PyCharm, etc.): open the integrated terminal, it's usually already pointing to the project folder
Mac: right-click the project folder → New Terminal at Folder
Windows: open the folder, click in the address bar, type cmd, press Enter

Activate the Python environment:

source venv/bin/activate    # Mac / Linux
venv\Scripts\activate       # Windows

Step 4: Start the app to verify

streamlit run app.py

The app detects that the CSV has changed and automatically runs the update script before loading. Check the terminal output: it should say Data files updated successfully. Then browse the app to confirm the change looks correct.

If the auto-update doesn't seem to work (no message in the terminal), you can trigger it manually as a fallback: python data/update.py

Step 5: Commit and push

In the terminal:

git add -A -- ':!venv/'
git commit -m "Your description here (e.g. Update coefficients for Afforestation)"
git push

Hugging Face will detect the push and rebuild the Space automatically.

Quick reference: where is each parameter?

I want to change…	File	Where exactly
A coefficient value (min / median / max)	`data/inputs_aggregated.csv`	Columns `min`, `median`, `max`: find the row by `Implementation` + `Resource`
The name of a CDR method	`data/inputs_aggregated.csv`	Column `Implementation
Which group a CDR method belongs to	`data/inputs_aggregated.csv`	Column `Method group`
The name of a resource	`data/inputs_aggregated.csv`	Column `Resource`
Which group a resource belongs to	`data/inputs_aggregated.csv`	Column `Resource group`
The unit of a resource group	`data/inputs_aggregated.csv`	Column `Unit`: must be consistent across all rows for that group
Whether a resource is optional for a method	`data/inputs_aggregated.csv`	Column `Is secondary`: `TRUE` = optional, `FALSE` = always active
The display order of methods or resources	`data/inputs_aggregated.csv`	Row order: in the app, groups are displayed in the order their first row is encountered in the CSV; within a group, methods or resources are displayed in the order their rows appear. Move rows up/down to reorder
The app's UI colours (background, buttons, links)	`.streamlit/config.toml`	`primaryColor`, `backgroundColor`, `secondaryBackgroundColor`, `textColor`
The chart colours (Plotly graphs)	`config/config.py`	`CRRA_COLORS` dictionary
The app's UI font	`.streamlit/config.toml`	`font` - accepted values: `"sans-serif"`, `"serif"`, `"monospace"`
The chart font (Plotly graphs)	`config/config.py`	`CRRA_FONT` variable
The app's UI colours and font (fine-tuned overrides)	`assets/styles.css`	`:root` block at the top of the file - change the hex values next to each variable name
The tooltip text on a method name	`data/tooltips.xlsx`	Sheet Methods: column `Tooltip`
The tooltip text on a resource name	`data/tooltips.xlsx`	Sheet Resources: column `Tooltip`
UI spacing, font sizes, button styles	`assets/styles.css`	Standard CSS: search for the element name
Python library versions	`requirements.txt`	One line per library: see Library Maintenance Guide below

After any change to inputs_aggregated.csv, start the app - it detects the change and updates automatically. After changes to config/config.py, assets/styles.css, or data/tooltips.xlsx, simply restart the app.

File structure (what each file does)

File	What it is	Do you edit it?
`data/inputs_aggregated.csv`	Main data file: all methods, resources and coefficients	✅ Yes
`data/update.py`	Script that regenerates everything from the CSV	❌ No
`data/cost_sliders.json`	Auto-generated. Coefficients used by the app	❌ No (auto)
`data/method_groups.json`	Auto-generated. CDR method categories	❌ No (auto)
`data/secondary_resources.json`	Auto-generated. optional resource assignments	❌ No (auto)
`data/inputs_template.xlsx`	Auto-generated. Excel template for users	❌ No (auto)
`data/resource_template.xlsx`	Auto-generated. Excel template for users	❌ No (auto)
`data/tooltips.xlsx`	Tooltip texts shown on hover in the app	✅ Yes

⚠️ Never edit the auto-generated files directly: your changes will be overwritten the next time update.py is run.

Understanding `inputs_aggregated.csv`

Each row represents one resource used by one CDR method, with the following columns:

Column	Description	Example
`Implementation`	Name of the CDR method	`Afforestation`
`Method group`	Category the method belongs to	`Forestry`
`Resource`	Name of the resource used	`Other land`
`Resource group`	Category the resource belongs to	`Land`
`Type`	Physical type of the resource	`Physical`
`Unit`	Unit of the coefficient (per MtCO₂)	`Mha/MtCO2`
`min`	Minimum coefficient value	`0.002`
`median`	Default (median) coefficient value	`0.094927`
`max`	Maximum coefficient value	`0.384615`
`Is secondary`	Whether this resource is optional for this method	`FALSE`

Primary resources (Is secondary = FALSE) are always active in the model. Secondary resources (Is secondary = TRUE) are optional: users can toggle them on in the app.

Maintenance cases

Before you start

Each case below covers two workflows:

Option 1 - Hugging Face: edit directly in the browser, no local setup needed.
Option 2 - Local: make changes on your machine, then push to Hugging Face.

For Option 2, make sure the project is cloned and the Python environment is set up (see Option 2 at the top of this guide). Before running any command in the terminal, activate the environment:

source venv/bin/activate    # Mac / Linux
venv\Scripts\activate       # Windows

You should see (venv) at the start of your terminal line.

For all Option 2 cases below: after editing the CSV, start the app (streamlit run app.py) to verify - it detects the change and runs the update automatically. If the auto-update doesn't trigger (no message in the terminal), run python data/update.py manually as a fallback.

Case 1 - Modify existing coefficient values

When to use: The literature has been updated and you want to change the min/median/max values for an existing method-resource pair.

Option 1 - On Hugging Face

In the Space, click the Files tab.
Open data/inputs_aggregated.csv → click the pencil icon to edit inline.
Find the row with the correct Implementation and Resource. Edit the min, median, and/or max columns.
Scroll down → Commit directly to main branch → Commit changes to main.
The Space restarts. Click Logs to confirm: Data files updated successfully.

Option 2 - Locally

Open data/inputs_aggregated.csv in Excel or a text editor.
Find the row with the correct Implementation and Resource.
Edit the min, median, and/or max columns.
Save the file, then start the app to verify:
```
streamlit run app.py
```

Commit and push:

git add -A -- ':!venv/'
git commit -m "Update coefficients for [method name] and [resource name]"
git push

Case 2 - Add an existing resource to an existing method

When to use: A CDR method should now also use a resource that is already tracked for other methods (e.g., water is already listed for other methods, and you want to add it to Afforestation too). If the resource doesn't exist at all yet, use Case 4 first.

Option 1 - On Hugging Face

In the Space, click Files → open data/inputs_aggregated.csv → click the pencil icon.
Add a new row, making sure to include all the expected columns (see Understanding inputs_aggregated.csv).
Commit directly to main → Space restarts. Check Logs for Data files updated successfully.

Option 2 - Locally

Open data/inputs_aggregated.csv and add a new row (see Understanding inputs_aggregated.csv).
Save the file, then start the app to verify:
```
streamlit run app.py
```

Commit and push:

git add -A -- ':!venv/'
git commit -m "Add [resource name] to [method name]"
git push

Case 3 - Add a new CDR method

When to use: A new carbon removal approach needs to be added to the tool.

Option 1 - On Hugging Face

In the Space, click Files → open data/inputs_aggregated.csv → click the pencil icon.
Add one row per resource the new method uses (see Understanding inputs_aggregated.csv). Use an existing Method group or create a new one.
Place the rows where you want the method to appear: display order follows row order in the CSV.
Commit directly to main → Space restarts. Check Logs for Data files updated successfully.

Option 2 - Locally

Open data/inputs_aggregated.csv.
Add one row per resource used by the new method. Fill in all columns (see Understanding inputs_aggregated.csv).
Save the file, then start the app to verify:
```
streamlit run app.py
```

Commit and push:

git add -A -- ':!venv/'
git commit -m "Add new CDR method: [method name]"
git push

Case 4 - Add a new resource

When to use: A resource that doesn't exist yet needs to be used (e.g., a new type of mineral or energy).

Option 1 - On Hugging Face

Decide which Resource group the new resource belongs to. If it belongs to an existing group, use the exact same name. If it needs its own group, see Case 5.
In the Space, click Files → open data/inputs_aggregated.csv → pencil icon.
Add one row per method that uses this resource (see column descriptions in Case 2).
Check the unit: all resources in a group must share the same unit (e.g., all Land resources use Mha/MtCO2). If the unit differs, create a new group (Case 5).
Commit directly to main → Space restarts. Check Logs.

Option 2 - Locally

Decide which Resource group the new resource belongs to.
Open data/inputs_aggregated.csv and add one row per method that uses this resource.
Verify the unit is consistent with the rest of the group.
Save, then start the app to verify:
```
streamlit run app.py
```

Commit and push:

git add -A -- ':!venv/'
git commit -m "Add new resource: [resource name]"
git push

Case 5 - Add a new resource group (new category)

When to use: Your new resource doesn't fit any existing category.

Option 1 - On Hugging Face

Same steps as Case 4, but write a new group name (never used before) in the Resource group column. The app will create a new section for it automatically. Place the rows where you want the group to appear: it appears at the position of its first row in the CSV.

Option 2 - Locally

Same steps as Case 4, but write a new group name in the Resource group column. The new group appears in the app at the position of its first row in the CSV.

Case 6 - Change whether a resource is optional (secondary) for a method

When to use: You want to make a resource optional (user can enable/disable it) or make it always required.

Option 1 - On Hugging Face

In the Space, click Files → open data/inputs_aggregated.csv → pencil icon.
Find the row with the correct Implementation and Resource.
Change Is secondary to TRUE (optional) or FALSE (always active).
Commit directly to main → Space restarts. Check Logs.

Option 2 - Locally

Open data/inputs_aggregated.csv.
Find the row with the correct Implementation and Resource.
Change Is secondary to TRUE (optional) or FALSE (always active).
Save, then start the app to verify:
```
streamlit run app.py
```

Commit and push:

git add -A -- ':!venv/'
git commit -m "Update secondary status for [resource] in [method]"
git push

Case 7 - Rename a method or resource

When to use: Correcting a spelling mistake or updating nomenclature.

⚠️ If users have saved scenario files (.xlsx or .zip) that reference the old name, those files will no longer load correctly after the rename.

Option 1 - On Hugging Face

Editing inline is not recommended for a rename because Find & Replace is not available in the browser editor. Instead:

Download data/inputs_aggregated.csv from the Files tab.
Open it in Excel. Use Find & Replace (Ctrl+H / Cmd+H) to replace the old name with the new name in the Implementation / Resource columns and, if the group name changed, in Method group / Resource group.
Also download data/tooltips.xlsx, update the name in the Method or Resource column of the relevant sheet, and save.
Upload both files via Files → data/ folder → + Contribute → Upload Files → Commit.
Space restarts → check Logs.

Option 2 - Locally

Open data/inputs_aggregated.csv. Use Find & Replace (Ctrl+H / Cmd+H) to replace the old name everywhere in the file.
Open data/tooltips.xlsx and update the name in the Method or Resource column of the relevant sheet. Save.
Start the app to verify:
```
streamlit run app.py
```

Commit and push:

git add -A -- ':!venv/'
git commit -m "Rename [old name] to [new name]"
git push

Case 8 - Edit tooltip texts (hover descriptions)

When to use: You want to add, update, or remove the short description that appears when a user hovers over a resource or method name in the app.

⚠️ The name in column Resource or Method must match exactly the name used in inputs_aggregated.csv (same spelling, same capitalisation). If you rename a method or resource in the CSV, remember to update the name in tooltips.xlsx too.

Option 1 - On Hugging Face

The .xlsx file cannot be edited inline on Hugging Face. Instead:

In the Files tab, open the data/ folder → download tooltips.xlsx.
Open it in Excel. Go to the relevant sheet (Resources or Methods).
Find the row with the name you want to edit. Fill in or change the Tooltip column. Leave it empty to remove a tooltip.
Save the file.
Back in the Files tab, go to data/ → + Contribute → Upload Files.
Upload your edited tooltips.xlsx → Commit directly to main.
The Space restarts. No update script runs for tooltips - the file is read directly by the app on each restart.

Option 2 - Locally

Open data/tooltips.xlsx in Excel.
Go to the relevant sheet (Resources or Methods).
Find the row with the name you want to edit. Fill in or change the Tooltip column. Leave it empty to remove a tooltip.
Save the file.
Restart the app (Ctrl+C then streamlit run app.py). No need to run update.py.

Commit and push:

git add data/tooltips.xlsx
git commit -m "Update tooltip texts"
git push

Case 10 - Change the display order of methods or resources

When to use: You want a method or resource group to appear earlier or later in the app (Resource inputs, Method constraints tabs...).

Option 1 - On Hugging Face

Moving rows is easier in Excel than in a browser editor:

Download data/inputs_aggregated.csv from the Files tab.
Open it in Excel. Move the relevant rows up or down - the app reads the CSV from top to bottom.
Upload the updated file via Files → data/ → + Contribute → Upload Files → Commit.
Space restarts → check Logs for Data files updated successfully.

Option 2 - Locally

Open data/inputs_aggregated.csv.
Move the relevant rows up or down.
Save, then start the app to verify:
```
streamlit run app.py
```

Commit and push:

git add -A -- ':!venv/'
git commit -m "Reorder [methods/resources]"
git push

After any change - checklist

Option 1 (Hugging Face direct):

Edited and committed inputs_aggregated.csv on Hugging Face
Checked the Space logs: confirmed Data files updated successfully.
Verified the change appears correctly in the app

Option 2 (local):

Saved inputs_aggregated.csv
Started the app locally (streamlit run app.py): confirmed Data files updated successfully. in the terminal
Verified the change appears correctly in the app
Committed all modified files (CSV, JSON files, templates) and pushed

Library Maintenance Guide

This section explains how to keep the tool's software dependencies up to date, safely and without breaking anything.

What is a library (dependency)?

The tool is built on top of external software packages ("libraries") that handle things like displaying the interface, reading Excel files, or running the optimization. These libraries are listed in requirements.txt with their exact version numbers (e.g., streamlit==1.45.1).

Libraries sometimes release updates - fixing bugs, patching security holes, or adding new features. The risk is that an update can also change how a library behaves and break the tool.

The key libraries and what they do

These are the libraries that matter most for this tool:

Library	Role	Risk if updated
streamlit	Powers the entire web interface (tabs, sliders, buttons, charts) - currently 1.58.0	🔴 High - UI layout and widget behavior can change
streamlit-option-menu	The navigation sidebar menu	🟡 Medium - must stay compatible with streamlit version
pandas	Reads and processes CSV and Excel files	🟡 Medium - occasional API changes
numpy	Numerical operations (arrays, math)	🟡 Medium - used by scipy and pandas
scipy	Runs the LP optimization (`linprog`)	🟡 Medium - optimization results could change
plotly	All charts and interactive graphs	🟡 Medium - chart API evolves
openpyxl	Reads and writes Excel files (.xlsx)	🟢 Low - stable API
pillow	Image handling (indirect use)	🟢 Low
requests / urllib3 / certifi	Network and security	🟢 Low - update freely for security patches

Dependency chain (what depends on what)

Some libraries require specific versions of others to work. Updating one can force updates to others:

streamlit ──────────────► tornado (web server)
            ├───────────► altair (charts)
            ├───────────► pyarrow (data performance)
            ├───────────► narwhals (data compatibility)
            ├───────────► protobuf
            └───────────► cachetools, blinker, click, toml

pandas ─────────────────► numpy
        ├───────────────► pyarrow
        └───────────────► python-dateutil, pytz, tzdata

scipy ──────────────────► numpy

plotly ─────────────────► narwhals

openpyxl ──────────────► pillow

Practical implication: if you update streamlit, you may also need to update tornado, altair, pyarrow, etc. If you update pandas, you may need to update numpy. These cascading updates are handled automatically by pip (see below), but they can introduce unexpected changes.

When to update

✅ Update immediately

pip-audit flags a vulnerability - run pip-audit regularly (see Check for vulnerabilities); any flagged library should be patched without delay.
Security vulnerabilities in requests, urllib3, certifi, or pillow - these are reported publicly and should be patched fast even before pip-audit catches them.
Check https://pypi.org/project/[library-name] or subscribe to GitHub security alerts.

✅ Update when needed

Streamlit - when a new feature in the app requires it, or when a critical bug in the current version is causing problems.
pandas / numpy / scipy - when a bug in the current version is causing incorrect data processing or optimization results.
openpyxl - when Excel import/export breaks with newer .xlsx files.

❌ Do NOT update just because a newer version exists

"Latest is always better" is not true for production tools. An update that works fine in isolation can break interactions between libraries.
The tool has been tested with the versions in requirements.txt. Changing them without testing is risky.

Checking for outdated libraries

To see which installed libraries have newer versions available:

pip list --outdated

This shows a list with current and latest versions. Use this to identify what could be updated but don't update everything at once.

How to safely update a library

Never update all libraries at once. Update one at a time, test the tool after each one.

Step 1 - Back up the current working state

cp requirements.txt requirements.txt.backup

Step 2 - Simulate first (dry run)

Before applying anything, check whether the update would cause conflicts:

pip install --upgrade streamlit --dry-run

If pip reports conflicts, investigate before proceeding.

Step 3 - Update the library

pip install --upgrade streamlit

Or to a specific version:

pip install streamlit==1.58.0

Step 4 - Verify no dependencies are broken

pip check

This confirms that all installed packages are mutually compatible. If it reports any conflict, do not proceed to the next step - restore the backup instead (see Step 6).

Step 5 - Freeze the new versions to requirements.txt

pip freeze > requirements.txt

pip freeze lists every library currently installed in the environment with its exact version number, and > requirements.txt writes that list into the file. This ensures that anyone who installs from requirements.txt later, including Hugging Face when the Space rebuilds, gets exactly the same versions you just tested.

Step 6 - Test the tool

Start the app: streamlit run app.py
Go through every tab
Import and export an Excel file
Run an optimization
Check that charts display correctly

Step 7 - If something breaks

Restore the backup:

pip install -r requirements.txt.backup
cp requirements.txt.backup requirements.txt

Step 8 - If everything works

Delete the backup file, it's no longer needed:

rm requirements.txt.backup

Then commit the updated requirements.txt and push.

How to install all libraries on a new machine

python -m venv venv
source venv/bin/activate        # Mac/Linux
# or: venv\Scripts\activate     # Windows
pip install -r requirements.txt

Checking for known security vulnerabilities

Install the security scanner:

pip install pip-audit

Run it:

pip-audit

This will flag any library with a known security issue and suggest a fix. These should be addressed promptly.

Recommended frequency: run pip-audit at least once a year, even if the tool appears stable. Security vulnerabilities are discovered independently of whether the library has changed - a version that was safe last year may be flagged today.

Using an AI assistant to handle updates

Library updates sometimes require code changes alongside the pip update - for example, a deprecated function that was renamed, or a new argument that became required. Doing this manually requires reading changelogs, finding every affected line, and testing the result.

An AI coding assistant such as Claude Code can handle this end-to-end:

Run pip-audit or pip list --outdated and paste the output into Claude Code
Describe the update you want to make (e.g., "update pandas and fix any deprecation warnings")
Claude Code will update the library, search the codebase for affected code, apply the necessary fixes, and update requirements.txt

This is especially useful when:

A deprecation warning points to a function that is used in multiple places across the codebase
A major version bump changes how a library behaves (e.g., pandas 3.x changing its default string backend)
You are not comfortable navigating the code yourself

The AI handles the mechanical parts; you remain responsible for testing the result with streamlit run app.py before committing and pushing.

Summary: update strategy

Security fix needed?  ──► Update immediately (requests, urllib3, certifi, pillow)
Something is broken?  ──► Update only the relevant library, test thoroughly
Nothing is broken?    ──► Do not update

Carbon Dioxide Removal Optimizer

Table of contents

Data Maintenance Guide

How it works (overview)

Option 1: Edit directly on Hugging Face (no local setup needed)

Option 2: Work locally

If you do not already have git installed: - Mac: install it via https://git-scm.com/download/mac - Windows: install it via https://git-scm.com/download/win

Quick reference: where is each parameter?

File structure (what each file does)

Understanding inputs_aggregated.csv

Maintenance cases

Before you start

Case 1 - Modify existing coefficient values

Case 2 - Add an existing resource to an existing method

Case 3 - Add a new CDR method

Case 4 - Add a new resource

Case 5 - Add a new resource group (new category)

Case 6 - Change whether a resource is optional (secondary) for a method

Case 7 - Rename a method or resource

Case 8 - Edit tooltip texts (hover descriptions)

Case 10 - Change the display order of methods or resources

After any change - checklist

Library Maintenance Guide

What is a library (dependency)?

The key libraries and what they do

Dependency chain (what depends on what)

When to update

✅ Update immediately

✅ Update when needed

❌ Do NOT update just because a newer version exists

Checking for outdated libraries

How to safely update a library

How to install all libraries on a new machine

Checking for known security vulnerabilities

Using an AI assistant to handle updates

Summary: update strategy

Understanding `inputs_aggregated.csv`