Update guidelines.md

guidelines.md

@@ -1,28 +1,14 @@
 ## Overview
 
-This document provides guidelines for evaluating the fluency of Norwegian responses generated by language models. Annotators will compare pairs of responses (Response A and Response B) and determine which response demonstrates better fluency, or if they are equally fluent. The evaluation focuses exclusively on language quality, naturalness, and grammaticality.
+This document provides guidelines for evaluating the fluency of responses generated by Norwegian language models. Annotators will compare pairs of responses (Response A and Response B) and determine which response demonstrates better fluency, or if they are equally fluent.
 
-## Key principle
-
-**Fluency evaluation is strictly limited to linguistic quality.** Do NOT consider:
-- Factual accuracy or correctness
-- Completeness of information
-- Creativity or originality
-- Formatting or structure (unless it affects readability)
-- Length or conciseness
+The evaluation focuses exclusively on language quality, naturalness, and grammaticality. Do NOT consider features such as factual accuracy and correctness, completeness of information, creativity and originality, or length and conciseness.
 
 ## Definitions
 
 ### What is fluency?
 
-Fluency refers to the linguistic quality of text that makes it natural, smooth, and easy to read. A fluent response:
-
-- **Grammatically correct**: Follows standard grammar rules with proper syntax
-- **Natural-sounding**: Reads like something a native speaker would write
-- **Coherent**: Maintains logical flow between sentences and paragraphs
-- **Well-formed**: Uses appropriate vocabulary, punctuation, and sentence structure
-- **Smooth**: Flows naturally without awkward phrasing or jarring transitions
-- **Norwegian**: The models respond to Norwegian prompts and so they should always be either in Norwegian Bokmål or Norwegian Nynorsk
+Fluency refers to the linguistic quality of text that makes it natural, smooth, and easy to read. It should look like a text written by a native speaker. A fluent text should consistently use either Bokmål or Nynorsk (depending on the prompt), and should sound genuinely Norwegian rather than as it were translated from another language.
 
 ### Fluency issues to look for
 
@@ -31,18 +17,18 @@ When evaluating fluency, pay attention to:
 1. **Grammar errors**: agreement errors (e.g. adjective-noun or determiner-noun disagreement), incorrect verb tense, incorrect word order (violating V2 requirement), wrong word forms
 2. **Awkward phrasing**: Unnatural word order, stilted expressions, robotic language
 3. **Punctuation problems**: Missing or incorrect punctuation that affects readability
-4. **Word choice issues**: Inappropriate vocabulary, incorrect word usage, repetitive language
+4. **Word choice issues**: Inappropriate vocabulary, incorrect word usage, repetitive language, wrong use of idioms or phrases, incorrect spacing of formation of compound words ("kaffe kopp" vs "kaffekopp"), preposition errors ("på" vs "i")
 5. **Flow disruptions**: Abrupt transitions, disconnected ideas within sentences
-6. **Spelling errors**: Typos and misspellings that affect readability
+6. **Spelling errors**: Typos and misspellings, wrong capitalization, incorrect use of diacritics (e.g. "å" vs "a", "ø" vs "o")
 7. **Translationese**: A common problem of language models is that they base their output on English -- the majority language in the language corpus. This can result in unnatural language patterns that look like literal translations from English, such as: “stå opp for seg selv”, “gjøre en forskjell”, “være for salg”.
 
 ## Annotation procedure
 
 ### Step-by-Step process
 
-1. **Read both responses completely** without making immediate judgments
-2. **Focus solely on language quality** - ignore content accuracy and relevance
-3. **Identify fluency issues** in each response using the criteria above
+1. **Read the prompt**: Do not analyze the fluency of the prompt, but look at it to understand the context and language style.
+2. **Read both responses completely** without making immediate judgments
+3. **Identify fluency issues** in each response using the criteria above, ignore content accuracy and relevance
 4. **Compare the severity and frequency** of fluency issues between responses
 5. **Make your decision** based on overall fluency
 
@@ -52,13 +38,13 @@ You must select one of three options:
 
 - **A is more fluent**: Response A has better overall language quality than Response B
 - **B is more fluent**: Response B has better overall language quality than Response A
-- **Equal fluency**: Both responses have similar language quality (minor differences that don't clearly favor either response)
+- **Equally fluent**: Both responses have similar language quality (minor differences that don't clearly favor either response)
 
 ### Important guidelines
 
 - **Minor differences matter**: Even small improvements in fluency should influence your decision
 - **Be consistent**: Apply the same standards across all evaluations
-- **When in doubt about equality**: If you cannot decisively determine which is better after careful analysis, select "Equal fluency"
+- **When in doubt about equality**: If you cannot decisively determine which is better after careful analysis, select "Equally fluent"
 
 ## Examples
 
@@ -82,6 +68,8 @@ TODO
 
 TODO
 
+**Other language than Norwegian**: If one of the responses is in a different language (e.g. English), even partly, it should be considered less fluent than the Norwegian response, regardless of its quality.
+
 **Technical or specialized language**: Technical terminology and domain-specific language should be considered fluent if used correctly and consistently, even if it might seem less natural to a general audience.
 
 **Formatting issues**: Ignore formatting differences (bold, italics, bullet points) unless they directly impact readability or sentence structure.