Trying to fix loading error. Note that I'm adding trust_remote_code=True; when new datasets are added, should be verified that this is OK.

Fixing loading error.

Fixing Spaces error so this runs again.

All changes minus temp.jsonl from last 3 commits

adding in "HuggingFaceM4/OBELICS" reference

miss match on variable names

try fix

can only select available splits

Update data_measurements/streamlit_utils.py

Merge branch 'main' of https://huggingface.co/spaces/huggingface/data-measurements-tool-2 into main

Handling for no words

catch missing text_dset

Merge branch 'main' of https://huggingface.co/spaces/huggingface/data-measurements-tool-2 into main

merge

cache and link

tokenized df bug

in deployment mode, check whethet the cache_dir exists

Removing option for c4 no clean

Try..except catching for errors

Merge branch 'main' of https://huggingface.co/spaces/huggingface/data-measurements-tool-2 into main

Adding another check to see if live before computing dset peek

Merge branch 'main' of https://huggingface.co/spaces/huggingface/data-measurements-tool-2 into main

Fixing some minor breaks

patch to show <10000 point in nPMI for performance

fix text in intro

fix length selectbox

Experimenting with direct merge conflict push to Spaces

Merge

Minor

Merging from rollback

merging dataset statistics file

Merging back dataset statistics

Be gone, you merge conflicting filegit rm data_measurements/dataset_statistics.py

Updating from rollback

Update from rollback

Adding dependencies for images

fix embeddings and load figure from file

Switching slider to selectbox for text lengths

Changing text lengths plot to a static one, saving to .png

Changing aggrid back to dataframe, even if we can't make the width dynamic

Change to npmi display ordering

Loading per-widget. Various changes to streamlit interactions for efficiency.

One more flag passing needed for setting live deployment

Adds flag for live deployment so that things will not be all recalculated when live.

Standardizing filenaming a bit.

More modularizing; npmi and labels

Some additional modularizing and caching of the text lengths widget

Modularization and caching of text length widget

Removes extraneous debugging print statements

Begins modularizing so that each widget can be independently loaded without having a requirement on the ordering of load_or_preparing in app.py. This means that each function corresponding to a widget will check if the variables it depends on have been calculated yet. If not, it will call back to calculate them. Because of the messiness this causes with passing the use_cache variable around, I've now set use_cache as a global variable, set when the DatasetStatisticsCacheClass is initialized, and removed the use_cache arguments appearing in nearly every function.