microsoft

/

aurora

like 48

Follow

Microsoft 17.4k

Pickled objects bring along some serious potential security risks, this adds the static fields as numpy files as an option to allow users to avoid calling pickle in downstream applications to load these fields.
Script used to generate:

import pickle
import numpy as np

for deg in ["0.25", "0.1"]:
    with open(f"../aurora-{deg}-static.pickle", "rb") as f:
        static_vars_ds = pickle.load(f)

    np.save(f"aurora-{deg}-static-z.npy", static_vars_ds["z"])
    np.save(f"aurora-{deg}-static-lsm.npy", static_vars_ds["lsm"])
    np.save(f"aurora-{deg}-static-slt.npy", static_vars_ds["slt"])

    assert np.allclose(np.load(f"aurora-{deg}-static-z.npy"), static_vars_ds["z"])
    assert np.allclose(np.load(f"aurora-{deg}-static-lsm.npy"), static_vars_ds["lsm"])
    assert np.allclose(np.load(f"aurora-{deg}-static-slt.npy"), static_vars_ds["slt"])

print("Pass")

Hey @NickGeneva ! Thanks for the suggestion. You're totally right. We should change the format for something safer. I'm thinking perhaps netCDF4, since that would still allow all arrays to be bundled together.

@mjstanley has uploaded netCDF4 versions of the Pickle files.