name: "🤗 Model Evaluation Request"
description: Request EvalPlus maintainers to evaluate your model independently and update it on our leaderboard.
title: "🤗 [REQUEST] - FILL_THE_MODEL_NAME_HERE"
labels: ["model eval"]
body:
  - type: textarea
    id: about
    attributes:
      label: "Model introduction"
      description: Provide a brief introduction to the model.
      placeholder: The models is created by ... and is used for ...
    validations:
      required: true
  - type: input
    id: url
    attributes:
      label: "Model URL"
      description: Indicate the URL (e.g., huggingface or other release pages) of the model
      placeholder: https://huggingface.co/[???]/[???]
    validations:
      required: true
  - type: textarea
    id: other
    attributes:
      label: "Additional information (Optional)"
      description: Special steps indicating how to run the model with preferably scripts/codes.
      placeholder: What data type precision should be used? What is the minimal hardware requirement? Can it be accelerated by tools such as vLLM?
    validations:
      required: false
  - type: textarea
    id: decomtamination
    attributes:
      label: "Decontamination"
      description: How does the authors avoid contamination for their training data?
      placeholder: Please clarify the decontamination steps and quantify it, e.g., N-gram match of ground-truth code in the training dataset.
    validations:
      required: true
  - type: dropdown
    id: author
    attributes:
      label: "Author"
      description: "Are you (one of) the author(s) of the model?"
      multiple: false
      options:
        - "Yes"
        - "No"
    validations:
      required: true
  - type: dropdown
    id: data
    attributes:
      label: "Data"
      description: "Is the training/fine-tuning data available in public?"
      multiple: false
      options:
        - "Yes (If so please specify in 'Additional information')"
        - "No"
    validations:
      required: true
  - type: checkboxes
    id: security
    attributes:
      label: "Security"
      options:
        - label: "I confirm that the model is safe to run which is not designed to produce malicious code or content."
          required: true
  - type: checkboxes
    id: integrity
    attributes:
      label: "Integrity"
      options:
        - label: "I confirm that the model comes from unique and original work and does not contain any plagiarism."
          required: true