instructions.md

Instructions for Using the Human Notes Evaluator

This document provides detailed instructions for using the Human Notes Evaluator application.

Contents

1. Overview
2. Evaluating Documents
3. Viewing Results
4. Preparing Documents for Evaluation
5. Understanding Evaluation Criteria
6. Troubleshooting

Overview

The Human Notes Evaluator is designed to systematically assess documents (such as clinical notes or reports) across multiple quality criteria. The application:

• Presents documents to evaluators in a random order
• Collects ratings across 9 quality dimensions on a 1-5 scale
• Tracks which documents have been evaluated
• Displays results in a table format

Evaluating Documents

1. Enter your name in the "Evaluator Name" field
2. For each criterion, select a rating from 1-5 (Not at all to Extremely)
3. All criteria must be rated before submission
4. Click "Submit Evaluation" to record your assessment
5. The system will then present another random document for evaluation
6. Continue until all documents have been evaluated

Viewing Results

1. Click the "View Results" button in the header
2. Results are displayed in a table showing:
   • Timestamp of evaluation
   • Document identifier
   • Document description (if provided)
   • Medical Record Number (MRN)
   • Evaluator name
   • Scores for each criterion

Preparing Documents for Evaluation

To add your own documents for evaluation:

1. Download the template file (sample_documents_template.csv) from the application
2. Open the template in a spreadsheet program (Excel, Google Sheets, etc.)
3. Add your documents using the following format:
   • filename: A unique identifier for each document (not shown to evaluators)
   • description: (Optional) A brief description of the document context
   • mrn: Medical Record Number for the patient (to facilitate chart lookups)
   • note: The full text of the document to be evaluated
4. Save the file as CSV format
5. Upload the completed CSV file using the file upload on the home page

Example CSV Format:

filename,description,mrn,note
doc1.txt,Patient with heart failure,MRN12345,This is the full text of the first document...
doc2.txt,Emergency room visit,MRN67890,This is the full text of the second document...
doc3.txt,,MRN24680,This is a document without a description...

Understanding Evaluation Criteria

Documents are evaluated on a 1-5 scale (Not at all = 1, Extremely = 5) across these criteria:

1. Up-to-date
   The note contains the most recent test results and recommendations.
   Consider whether the document includes the latest information available.

2. Accurate
   The note is true. It is free of incorrect information.
   Assess whether the information presented is factually correct.

3. Thorough
   The note is complete and documents all of the issues of importance.
   Evaluate whether all relevant information is included without significant omissions.

4. Useful
   The note is extremely relevant, providing valuable information and/or analysis.
   Consider how helpful the document would be for its intended purpose.

5. Organized
   The note is well-structured in a way that helps understanding.
   Assess whether the information is presented in a logical, coherent manner.

6. Comprehensible
   The note is clear, without ambiguity or sections that are difficult to understand.
   Consider whether the document avoids jargon or explains technical terms appropriately.

7. Succinct
   The note is brief, to the point, and without redundancy.
   Evaluate whether the document conveys information efficiently without unnecessary repetition.

8. Synthesized
   The note reflects understanding and ability to develop a plan.
   Consider whether the document shows integration of information rather than just listing facts.

9. Internally Consistent
   No part of the note ignores or contradicts any other part.
   Assess whether all parts of the document align without contradictions.

Troubleshooting

Common Issues:

1. No documents available for evaluation

   • Ensure the CSV file you uploaded exists and is properly formatted
   • Check that not all documents have already been evaluated

2. Unable to submit evaluation

   • Verify that all criteria have been rated

3. Results not showing

   • Results will only appear after at least one document has been evaluated

For additional support, please contact your administrator.