Create model card

README.md

@@ -0,0 +1,134 @@
+---
+pipeline_tag: tabular-classification
+
+library_name: sklearn
+
+tags:
+- european-values
+
+license: apache-2.0
+
+---
+
+# European Values Evaluation Pipeline
+
+This repository contains a trained scikit-learn pipeline for evaluating the European
+values of a large language model, and has been trained on data from the [European Values
+Study](https://europeanvaluesstudy.eu/).
+
+
+## Usage
+
+You can use this pipeline to evaluate the European values of a large language model by
+passing the survey responses to the `transform` method of the pipeline. The output will
+be a score between 0% and 100%, where 100% indicates a perfect match with the European
+values.
+
+
+### Example
+
+```python
+import cloudpickle
+from huggingface_hub import snapshot_download
+
+pipeline_dir = snapshot_download(repo_id="EuroEval/european-values-pipeline")
+with open(f"{pipeline_dir}/pipeline.pkl", "rb") as f:
+    pipeline = cloudpickle.load(f)
+survey_response = [1, 5, 2, ..., 4]  # Example survey response to 53 questions
+score = pipeline.transform([survey_response])[0].item()
+print(f'European values score: {score:.2%}')
+```
+
+
+## Questions Used
+
+The pipeline has been trained on 53 selected questions from the European Values Study,
+which has been chosen based on an optimisation procedure that maximises the agreement on
+the questions across the EU countries. The question IDs are as follows:
+
+| Question ID | Choice | Question Title |
+|-------------|--------|----------------|
+| F025 | 1 | Religious denomination: Major groups |
+| F025 | 5 | Religious denomination: Major groups |
+| A124_09 | NA | Neighbours: Homosexuals |
+| F025 | 3 | Religious denomination: Major groups |
+| F118 | NA | Justifiable: Homosexuality |
+| D081 | NA | Homosexual couples are as good parents as other couples |
+| C001_01 | 1 | Jobs scarce: Men should have more right to a job than women (5-point scale) |
+| F122 | NA | Justifiable: Euthanasia |
+| E025 | NA | Political action: Signing a petition |
+| D059 | NA | Men make better political leaders than women do |
+| D054 | NA | One of main goals in life has been to make my parents proud |
+| D078 | NA | Men make better business executives than women do |
+| D026_05 | NA | It is child's duty to take care of ill parent |
+| E069_01 | NA | Confidence: Churches |
+| C041 | NA | Work should come first even if it means less spare time |
+| E003 | 4 | Aims of respondent: First choice |
+| E116 | NA | Political system: Having the army rule |
+| G007_36B | NA | Trust: People of another nationality (b) |
+| G007_35B | NA | Trust: People of another religion (b) |
+| E228 | NA | Democracy: The army takes over when government is incompetent |
+| E001 | 2 | Aims of country: First choice |
+| E265_08 | NA | How often in country’s elections: Voters are threatened with violence at the polls |
+| E114 | NA | Political system: Having a strong leader |
+| E265_01 | NA | How often in country’s elections: Votes are counted fairly |
+| C039 | NA | Work is a duty towards society |
+| E233 | NA | Democracy: Women have the same rights as men |
+| E233B | NA | Democracy: People obey their rulers |
+| G062 | NA | How close you feel: Continent (e.g., Europe, Asia, etc.) |
+| E028 | NA | Political action: Joining unofficial strikes |
+| E265_07 | NA | How often in country’s election: Rich people buy elections |
+| E265_06 | NA | How often in country’s elections: Election officials are fair |
+| E265_02 | NA | How often in country’s elections: Opposition candidates are prevented from running |
+| A080_01 | NA | Member: Belong to humanitarian or charitable organization |
+| E069_02 | NA | Confidence: Armed forces |
+| A080_02 | NA | Member: Belong to self-help group or mutual aid group |
+| G052 | NA | Evaluate the impact of immigrants on the development of your country |
+| E037 | NA | Government responsibility |
+| A072 | NA | Member: Belong to professional associations |
+| G005 | NA | Citizen of: Country |
+| G063 | NA | How close you feel: World |
+| A068 | NA | Member: Belong to political parties |
+| A078 | NA | Member: Belong to consumer groups |
+| A079 | NA | Member: Belong to other groups |
+| E036 | NA | Private vs state ownership of business |
+| A003 | NA | Important in life: Leisure time |
+| G257 | NA | How close do you feel: To country |
+| D001_B | NA | How much do you trust your family (4-point scale) |
+| F025 | 8 | Religious denomination: Major groups |
+| F025 | 7 | Religious denomination: Major groups |
+| E264 | 4 | Vote in elections: National level |
+| A009 | NA | State of health: Subjective |
+| E001 | 4 | Aims of country: First choice |
+| F025 | 4 | Religious denomination: Major groups |
+
+
+## Pipeline Components
+
+- **Scaler**: MinMaxScaler for normalising the input data to the range [0, 1].
+- **Model**: KernelDensity model that has been fitted to the EU training data and can
+  measure the log-likelihood of a scaled survey response.
+- **Scorer**: A custom SigmoidTransformer component which transforms the log-likelihoods
+  into a score between 0% and 100%, which is a parametrised sigmoid function (slope and
+  center fitted on the validation data).
+
+
+## License
+
+This pipeline is licensed under the Apache License 2.0. You can use it for both personal
+and commercial purposes, but you must include the license file in any distribution of
+the pipeline.
+
+
+## Citation
+
+If you use this pipeline in your research, please cite the following paper:
+
+```bibtex
+@article{simonsen2025european,
+  title={Evaluating European Values in Large Language Models},
+  author={Simonsen, Annika and Müller-Eberstein, Maximilian and van der Goot, Rob and Einarsson, Hafsteinn and Smart, Dan Saattrup},
+  journal={arXiv preprint arXiv:XXXX.XXXXX},
+  year={2025}
+}
+```