Spaces:
Running on CPU Upgrade
Update dabstep_benchmark/leaderboard.py
The submission validator rejects valid JSONL files with:
"Columns with non-string data type: task_id, agent_answer".
The issue seems to be in dabstep_benchmark/leaderboard.py:
submission_df = pd.read_json(submission_path, lines=True, dtype=str)
non_string_columns = [col for col in submission_df.columns if submission_df[col].dtype != 'object']
In newer pandas / infer_string mode, string columns have dtype "str" or StringDtype instead of object, so valid string columns are rejected.
Suggested fix:
from pandas.api.types import is_string_dtype
non_string_columns = [
col for col in submission_df.columns
if not is_string_dtype(submission_df[col])
]
@frisokingma @jeanmarcs @drublackberry @martinigoyanes @antonioramos @MindyKasting @davidlever @rokpopov @JorgeZapa @AaronAtAdyen @andreumora @KoenRoelofs @hannav @sergioadyen @zoranaAtadyen @wolfsinemm @BelleB @moktay @lchumaceiro @olgakostinaadyen @robertAdyen @tomjadams
Can someone from this company, for the sake of kindness and respectfullnes, let us know that if this benchmark is maintained at all? If not, that's fine, just let us know so that we don't need to waste our time.
@frisokingma @jeanmarcs @drublackberry @martinigoyanes @antonioramos @MindyKasting @davidlever @rokpopov @JorgeZapa @AaronAtAdyen @andreumora @KoenRoelofs @hannav @sergioadyen @zoranaAtadyen @wolfsinemm @BelleB @moktay @lchumaceiro @olgakostinaadyen @robertAdyen @tomjadams @iadyen @eggie5-adyen @martinigoyanes @andreumora
Can someone from this company, for the sake of kindness and respectfullnes, let us know that if this benchmark is maintained at all? If not, that's fine, just let us know so that we don't need to waste our time.