docs/benchmark_tasks/NanoCodeRAG/NanoCodeRAGOnlineTutorials.md

NanoCodeRAG / NanoCodeRAGOnlineTutorials

Overview

CodeRAG-Bench includes online tutorials as a retrieval source for code-generation support, collecting pages from sites such as GeeksforGeeks, W3Schools, Tutorialspoint, and Towards Data Science through ClueWeb22. In this Nano split, short tutorial or programming-problem titles must retrieve long articles containing prose, steps, code snippets, examples, dates, and explanations. The retrieval problem is to connect a developer's title-like need to the tutorial page that explains the relevant API, language feature, algorithm, or task.

Details

What the Original Data Measures

CodeRAG-Bench: Can Retrieval Augment Code Generation? collects online tutorials as one of five retrieval sources for code RAG. The paper says these tutorials come from sites including GeeksforGeeks, W3Schools, Tutorialspoint, and Towards Data Science, using raw HTML pages from ClueWeb22. Each page contains code snippets and textual explanations, spanning basic programming techniques through advanced library usage.

The paper's open retrieval analysis finds that tutorials can help some code generation settings, especially when the tutorial is about the same programming problem or gives detailed textual explanations. This Nano split isolates the tutorial retrieval source itself: the correct document is the tutorial article matching the query title or problem statement.

Observed Data Profile

The Nano split has 200 queries, 9,997 documents, and 200 positive qrel rows. Every query has one positive. Queries average 51.91 characters and are usually page-title style strings such as Android screen control, Linux secure deletion, C++ access modifiers, Python subscript printing, or GeeksforGeeks practice problem names. Documents average 5,722.55 characters, with many long tutorial pages that include dates, examples, code blocks, and step-by-step instructions.

The sampled documents are mostly article-like, not compact API references. They often contain boilerplate, multiple examples, and broad explanatory context. This means the retriever must use title and task intent while tolerating long documents that contain many unrelated tokens.

BM25 Difficulty

Using the dataset-provided BM25 candidate column, BM25 reaches nDCG@10 = 0.7472 and hit@10 = 0.8400. It ranks 132 positives first and finds 168 positives in the top 10. Lexical matching is strong because titles and tutorials often share exact phrases, framework names, and language names.

The failures show where exact matching is brittle. A query like java.time.LocalDate.atTime() Method Example can be outranked by unrelated web content, and Map in C++ Standard Template Library (STL) retrieves a nearby map operator page before the overview tutorial. Short or generic titles such as Tryit Editor v3.7 are especially ambiguous.

Training Data That May Help

Useful training data includes non-overlapping programming tutorial title-to-page pairs, developer search logs, Stack Overflow question-to-tutorial citations, documentation search data, and code-example retrieval where prose and snippets jointly explain the solution. Training should exclude the NanoCodeRAG tutorial evaluation queries, qrels, and positive tutorial pages.

Training should preserve long-document behavior. Models need to match the article's central task, not just an incidental code token that appears in a long page.

Synthetic Data Guidance

For document-to-question generation, use non-evaluation tutorial pages and generate title-like queries, "how to" questions, and practice-problem prompts that the tutorial answers. Preserve language names, framework names, function names, expected inputs, and output behavior.

For joint generation, create realistic tutorial pages with steps, examples, and code snippets, then generate concise page-title queries for them. Hard negatives should be tutorials from the same language or topic that solve a different task. Do not use Nano evaluation queries or positive tutorial pages as seeds.

Example Data

Query	Positive document
How to turn Android device screen on and off programmatically? (62 chars)	This example demonstrate about How to turn Android device screen on and off programmatically. Step 1 − Create a new project in Android Studio, go to File ⇒ New Project and fill all required details to create a new project. St ... [truncated 225 chars](6654 chars)
Tools to Securely Delete Files from Linux - GeeksforGeeks (57 chars)	16 Feb, 2021 Every time you delete a file from your Linux system using the shift + delete or rm command, it doesn’t actually permanently and securely delete the file from the hard disk. When you delete a file with the rm comm ... [truncated 225 chars](3940 chars)
Difference between Private and Protected in C++ with Example - GeeksforGeeks (76 chars)	03 Jan, 2022 Protected Protected access modifier is similar to that of private access modifiers, the difference is that the class member declared as Protected are inaccessible outside the class but they can be accessed by any ... [truncated 225 chars](2559 chars)
How to print Superscript and Subscript in Python? - GeeksforGeeks (65 chars)	24 Jan, 2021 Whenever we are working with formulas there may be a need of writing the given formula in a given format which may require subscripts or superscripts. There are several methods available to print subscripts and s ... [truncated 225 chars](2336 chars)
Maximum Difference | Practice | GeeksforGeeks (45 chars)	Given array A[] of integers, the task is to complete the function findMaxDiff which finds the maximum absolute difference between nearest left and right smaller element of every element in array.If the element is the leftmost ... [truncated 225 chars](12706 chars)

Dataset Information

Field	Value
Nano set	NanoCodeRAG
Backing dataset	NanoCodeRAG
Task / split	NanoCodeRAGOnlineTutorials
Hugging Face dataset	hakari-bench/NanoCodeRAG
Language	en
Category	code
Queries	200
Documents	9,997
Positive qrels	200
BM25 nDCG@10	0.7472
BM25 hit@10	0.8400
Query length avg chars	51.91
Document length avg chars	5,722.55

Public Sources

• CodeRAG-Bench: Can Retrieval Augment Code Generation?; 2025; Zora Zhiruo Wang, Akari Asai, Xinyan Velocity Yu, Frank F. Xu, Yiqing Xie, Graham Neubig, and Daniel Fried; DOI: 10.18653/v1/2025.findings-naacl.176.
• CodeRAG-Bench project page.
• CodeRAG-Bench GitHub repository.
• code-rag-bench/online-tutorials dataset card.

Hugging Face Links

• Nano dataset: hakari-bench/NanoCodeRAG
• Source dataset: code-rag-bench/online-tutorials

Source Reference Table

Title	Year	Type	URL
CodeRAG-Bench: Can Retrieval Augment Code Generation?	2025	arXiv paper	https://arxiv.org/abs/2406.14497
CodeRAG-Bench project page	2025	project page	https://code-rag-bench.github.io/
code-rag-bench/online-tutorials	2024	dataset card	https://huggingface.co/datasets/code-rag-bench/online-tutorials

Machine-Readable Metadata

benchmark_task_metadata:
  schema_version: 1
  document_status: first_pass
  nano_set: NanoCodeRAG
  backing_dataset: NanoCodeRAG
  dataset_id: hakari-bench/NanoCodeRAG
  task_name: NanoCodeRAGOnlineTutorials
  split_name: NanoCodeRAGOnlineTutorials
  language: en
  category: code
  document_path: docs/benchmark_tasks/NanoCodeRAG/NanoCodeRAGOnlineTutorials.md
  source_research:
    primary_source_type: benchmark_paper
    paper_pdf_or_html_checked: true
    paper_url: https://arxiv.org/abs/2406.14497
    additional_source_urls:
      - https://aclanthology.org/2025.findings-naacl.176/
      - https://code-rag-bench.github.io/
      - https://github.com/code-rag-bench/code-rag-bench
      - https://huggingface.co/datasets/code-rag-bench/online-tutorials
  counts:
    queries: 200
    documents: 9997
    positive_qrels: 200
  positives_per_query:
    average: 1.0
    min: 1
    median: 1.0
    max: 1
    multi_positive_queries: 0
    multi_positive_query_percent: 0.0
  text_stats_chars:
    query_mean: 51.905
    document_mean: 5722.545864
  bm25:
    ndcg_at_10: 0.7471740687
    hit_at_10: 0.84
    source: dataset_bm25_column
  learning:
    original_train_split: unknown
    evaluation_split_origin: CodeRAG-Bench online tutorials retrieval source sampled into NanoCodeRAG
    train_eval_overlap_audit: not_audited
    leakage_note: exclude NanoCodeRAG tutorial queries, qrels, and positive tutorial pages
    useful_training_data:
      - non-overlapping programming tutorial title-to-page pairs
      - developer search logs over tutorials and documentation
      - Stack Overflow question-to-tutorial citation pairs
      - code example retrieval with long tutorial hard negatives
    synthetic_data:
      document_generation: realistic programming tutorials with prose, steps, code snippets, inputs, outputs, and language-specific details
      question_generation: title-like and how-to programming queries grounded in the tutorial's main task
      answerability: the selected tutorial should explain the requested API, algorithm, or programming procedure
    multi_positive_training: single_positive_question_document_focus
  links:
    nano_dataset: https://huggingface.co/datasets/hakari-bench/NanoCodeRAG
    source_urls:
      - label: CodeRAG-Bench arXiv
        url: https://arxiv.org/abs/2406.14497
      - label: CodeRAG-Bench project page
        url: https://code-rag-bench.github.io/
      - label: CodeRAG-Bench GitHub
        url: https://github.com/code-rag-bench/code-rag-bench
      - label: code-rag-bench/online-tutorials
        url: https://huggingface.co/datasets/code-rag-bench/online-tutorials
    source_notes: []
  references:
    - title: "CodeRAG-Bench: Can Retrieval Augment Code Generation?"
      url: https://arxiv.org/abs/2406.14497
      year: 2025
      doi: 10.18653/v1/2025.findings-naacl.176
      is_paper: true
      source_confidence: definitive_paper_link