2308/2308.06385.md

Title: Responsible NLP Research Checklist

URL Source: https://arxiv.org/html/2308.06385

Markdown Content: Members of the ACL are responsible for adhering to the ACL code of ethics. The ARR Responsible NLP Research checklist is designed to encourage best practices for responsible research, addressing issues of research ethics, societal impact and reproducibility.

All supporting evidence can appear either in the main paper or the supplemental material. For each question, if you answer Yes, provide the section number; if you answer No, provide a justification.

Please do not modify, reorder, delete or add questions, question options or other wording of this document.

A For every submission

A1 Did you discuss the limitations of your work?

If you answer Yes, provide the section number; if you answer No, provide a justification.

{Form}\ChoiceMenu[height=0.3cm,width=0.3cm,bordercolor=,name=mainClaims,radio,radiosymbol=\ding 108,default=Yes]Yes,No,N/A\TextField[bordercolor=,name=mainClaimsJustification,value=We have included a section dedicated to Limitations after the Conclusions,multiline=true,height=4em, width=0.85] Section or Justification

A2 Did you discuss any potential risks of your work?

If you answer Yes, provide the section number; if you answer No, provide a justification.

{Form}\ChoiceMenu[height=0.3cm,width=0.3cm,bordercolor=,name=risks,radio,radiosymbol=\ding 108,default=Yes]Yes,No,N/A\TextField[bordercolor=,name=risksJustification,value=Risks are discussed in the Limitations section,multiline=true,height=4em, width=0.85] Section or Justification

A3 Do the abstract and introduction summarize the paper’s main claims?

If you answer Yes, provide the section number; if you answer No, provide a justification.

{Form}\ChoiceMenu[height=0.3cm,width=0.3cm,bordercolor=,name=abstractIntro,radio,radiosymbol=\ding 108,default=Yes]Yes,No,N/A\TextField[bordercolor=,name=abstractIntroJustification,value=The second last paragraphs of the Introduction discusses our contributions, and the abstract summarizes the claims,multiline=true,height=4em, width=0.85] Section or Justification

B Did you use or create scientific artifacts?

If you answer Yes, answer the questions below; if you answer No, you can skip the rest of this section.

{Form}\ChoiceMenu[height=0.3cm,width=0.3cm,bordercolor=,name=createArtifacts,radio,radiosymbol=\ding 108,default=Yes]Yes,No

If yes:

B1 Did you cite the creators of artifacts you used?

If you answer Yes, provide the section number; if you answer No, provide a justification.

{Form}\ChoiceMenu[height=0.3cm,width=0.3cm,bordercolor=,name=citeCreators,radio,radiosymbol=\ding 108,default=Yes]Yes,No,N/A\TextField[bordercolor=,name=citeCreatorsJustification,value=We cite the library trlx, which is the basis of the PPO optimization algorithm we use in Section 4.1. We also cite the sources for the pretrained language models we use through the text,multiline=true,height=4em, width=0.78] Section or Justification

B2 Did you discuss the license or terms for use and/or distribution of any artifacts?

If you answer Yes, provide the section number; if you answer No, provide a justification.

{Form}\ChoiceMenu[height=0.3cm,width=0.3cm,bordercolor=,name=legalGrounds,radio,radiosymbol=\ding 108,default=No]Yes,No,N/A\TextField[bordercolor=,name=legalGroundsJustification,value=All the previous artifacts we mention are licensed under open-source licences that allows research use, so we didn’t think it was necessary to include this in the main text,multiline=true,height=4em, width=0.78] Section or Justification

B3 Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)?

If you answer Yes, provide the section number; if you answer No, provide a justification.

{Form}\ChoiceMenu[height=0.3cm,width=0.3cm,bordercolor=,name=intendedUse,radio,radiosymbol=\ding 108,default=No]Yes,No,N/A\TextField[bordercolor=,name=intendedUseJustification,value=As we said in the previous response, all the artifacts we use have licenses that allow usage in research, so we didn’t think it was necessary to re-state this in the text,multiline=true,height=4em, width=0.78] Section or Justification

B4 Did you discuss the steps taken to check whether the data that was collected/used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it?

If you answer Yes, provide the section number; if you answer No, provide a justification.

{Form}\ChoiceMenu[height=0.3cm,width=0.3cm,bordercolor=,name=personallyIdentifiableInformationOrOffensiveContent,radio,radiosymbol=\ding 108,default=No]Yes,No,N/A\TextField[bordercolor=,name=personallyIdentifiableInformationOrOffensiveContentJustification,value=We don’t introduce new datasets, and the datasets we used do not disclose personal information,multiline=true,height=4em, width=0.78] Section or Justification

B5 Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.?

If you answer Yes, provide the section number; if you answer No, provide a justification.

{Form}\ChoiceMenu[height=0.3cm,width=0.3cm,bordercolor=,name=documentation,radio,radiosymbol=\ding 108,default=No]Yes,No,N/A\TextField[bordercolor=,name=documentationJustification,value=The experiments we perform are focused on English and on very narrow domains, so we believe this was not necessary,multiline=true,height=4em, width=0.78] Section or Justification

B6 Did you report relevant statistics like the number of examples, details of train/test/dev splits, etc. for the data that you used/created?

If you answer Yes, provide the section number; if you answer No, provide a justification.

{Form}\ChoiceMenu[height=0.3cm,width=0.3cm,bordercolor=,name=relevantStatistics,radio,radiosymbol=\ding 108,default=Yes]Yes,No,N/A\TextField[bordercolor=,name=relevantStatisticsJustification,value=These statistics can be reproduced going to the code we released, which is linked in the abstract of the paper,multiline=true,height=4em, width=0.78] Section or Justification

C Did you run computational experiments?

If you answer Yes, answer the questions below; if you answer No, you can skip the rest of this section.

{Form}\ChoiceMenu[height=0.3cm,width=0.3cm,bordercolor=,name=computationalExperiments,radio,radiosymbol=\ding 108,default=Yes]Yes,No

If yes:

C1 Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used?

If you answer Yes, provide the section number; if you answer No, provide a justification.

{Form}\ChoiceMenu[height=0.3cm,width=0.3cm,bordercolor=,name=reportReproducibility,radio,radiosymbol=\ding 108,default=Yes]Yes,No,N/A\TextField[bordercolor=,name=reportReproducibilityJustification,value=While we do not explicitly report the number of parameters, all the models we use are open-sourced so this can be checked at any time. Further experiment details can be found in Appendix B,multiline=true,height=4em, width=0.78] Section or Justification

C2 Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values?

If you answer Yes, provide the section number; if you answer No, provide a justification.

{Form}\ChoiceMenu[height=0.3cm,width=0.3cm,bordercolor=,name=bestFoundHyperparameter,radio,radiosymbol=\ding 108,default=Yes]Yes,No,N/A\TextField[bordercolor=,name=bestFoundHyperparameterJustification,value=See Appendix B and the code released, which is linked in the abstract,multiline=true,height=4em, width=0.78] Section or Justification

C3 Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run?

If you answer Yes, provide the section number; if you answer No, provide a justification.

{Form}\ChoiceMenu[height=0.3cm,width=0.3cm,bordercolor=,name=descriptiveStatistics,radio,radiosymbol=\ding 108,default=Yes]Yes,No,N/A\TextField[bordercolor=,name=descriptiveStatisticsJustification,value=The experiment results are in Section 4, and we report Tables and Figures with statistics such as standard deviations around the mean, or boxplots.,multiline=true,height=4em, width=0.78] Section or Justification

C4 If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)?

If you answer Yes, provide the section number; if you answer No, provide a justification.

{Form}\ChoiceMenu[height=0.3cm,width=0.3cm,bordercolor=,name=existingPackages,radio,radiosymbol=\ding 108,default=Yes]Yes,No,N/A\TextField[bordercolor=,name=existingPackagesJustification,value=For the libraries we used (trlx) and the pretrained models we test, their information is provided in Section 4, Appendix B and the released code.,multiline=true,height=4em, width=0.78] Section or Justification

D Did you use human annotators (e.g., crowdworkers) or research with human subjects?

If you answer Yes, answer the questions below; if you answer No, you can skip the rest of this section.

{Form}\ChoiceMenu[height=0.3cm,width=0.3cm,bordercolor=,name=hummanAnnotators,radio,radiosymbol=\ding 108,default=No]Yes,No

If yes:

D1 Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.?

If you answer Yes, provide the section number; if you answer No, provide a justification.

{Form}\ChoiceMenu[height=0.3cm,width=0.3cm,bordercolor=,name=fullTextInstructions,radio,radiosymbol=\ding 108,default=]Yes,No,N/A\TextField[bordercolor=,name=fullTextInstructionsJustification,value=,multiline=true,height=4em, width=0.78] Section or Justification

D2 Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)?

If you answer Yes, provide the section number; if you answer No, provide a justification.

{Form}\ChoiceMenu[height=0.3cm,width=0.3cm,bordercolor=,name=payment,radio,radiosymbol=\ding 108,default=]Yes,No,N/A\TextField[bordercolor=,name=paymentJustification,value=,multiline=true,height=4em, width=0.78] Section or Justification

D3 Did you discuss whether and how consent was obtained from people whose data you’re using/curating (e.g., did your instructions explain how the data would be used)?

If you answer Yes, provide the section number; if you answer No, provide a justification.

{Form}\ChoiceMenu[height=0.3cm,width=0.3cm,bordercolor=,name=consent,radio,radiosymbol=\ding 108,default=]Yes,No,N/A\TextField[bordercolor=,name=consentJustification,value=,multiline=true,height=4em, width=0.78] Section or Justification

D4 Was the data collection protocol approved (or determined exempt) by an ethics review board?

If you answer Yes, provide the section number; if you answer No, provide a justification.

{Form}\ChoiceMenu[height=0.3cm,width=0.3cm,bordercolor=,name=ethicsAmountSpent,radio,radiosymbol=\ding 108,default=]Yes,No,N/A\TextField[bordercolor=,name=ethicsAmountSpentJustification,value=,multiline=true,height=4em, width=0.78] Section or Justification

D5 Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?

If you answer Yes, provide the section number; if you answer No, provide a justification.

{Form}\ChoiceMenu[height=0.3cm,width=0.3cm,bordercolor=,name=annotator,radio,radiosymbol=\ding 108,default=]Yes,No,N/A\TextField[bordercolor=,name=annotatorJustification,value=,multiline=true,height=4em, width=0.78] Section or Justification

E Did you use AI assistants (e.g., ChatGPT, Copilot) in your research, coding, or writing?

If you answer Yes, answer the question below; if you answer No, you can skip the rest of this section.

{Form}\ChoiceMenu[height=0.3cm,width=0.3cm,bordercolor=,name=aiAssistants,radio,radiosymbol=\ding 108,default=Yes]Yes,No

E1 Did you include information about your use of AI assistants?

If you answer Yes, provide the section number; if you answer No, provide a justification.

{Form}\ChoiceMenu[height=0.3cm,width=0.3cm,bordercolor=,name=aiAssistantsInformation,radio,radiosymbol=\ding 108,default=Yes]Yes,No,N/A\TextField[bordercolor=,name=aiAssistantsInformationJustification,value=We used gpt-3.5-turbo to evaluate the generations in a experiment. This is clearly stated in Section 4.,multiline=true,height=4em, width=0.78] Section or Justification