Upload folder using huggingface_hub

architecture/index.html

@@ -649,7 +649,7 @@
 </tr>
 <tr>
 <td><strong>Classifier</strong></td>
-<td>Echo-DSRN-98M (<code>EchoForSequenceClassification</code>) — sub-millisecond CPU inference</td>
+<td>Echo-DSRN-98M (<code>EchoForSequenceClassification</code>) — fast CPU inference</td>
 </tr>
 <tr>
 <td><strong>Annotation DB</strong></td>

index.html

@@ -630,7 +630,7 @@
 
 <h1 id="openaire-ai-research-evaluator">OpenAIRE-AI-Research-Evaluator</h1>
 <p>A multi-model LLM-as-Judge pipeline that builds annotation datasets to train <a href="https://github.com/ethicalabs-ai/Echo-DSRN">Echo-DSRN</a> intent classifiers — entry for the <a href="https://innovation.openaire.eu/component/content/article/openaire-ai-hackathon.html">OpenAIRE AI Hackathon 2026</a>, co-organised by OpenAIRE and Alien Intelligence.</p>
-<p>The app streams scientific paper metadata from the <a href="https://graph.openaire.eu">OpenAIRE Graph API</a>, classifies research intent with a 98M-parameter <a href="https://www.ethicalabs.ai/research/echo-dsrn/">Echo-DSRN</a> model (sub-millisecond CPU inference), and enlists multiple LLMs as judges to build a golden consensus dataset.</p>
+<p>The app streams scientific paper metadata from the <a href="https://graph.openaire.eu">OpenAIRE Graph API</a>, classifies research intent with a 98M-parameter <a href="https://www.ethicalabs.ai/research/echo-dsrn/">Echo-DSRN</a> model (fast CPU inference), and enlists multiple LLMs as judges to build a golden consensus dataset.</p>
 <h2 id="features">Features</h2>
 <ul>
 <li><strong>OpenAIRE Stream</strong> — live publication metadata via the OpenAIRE Graph API</li>

inference/index.html

@@ -717,7 +717,7 @@
 </tr>
 <tr>
 <td>Inference</td>
-<td>Sub-millisecond on CPU</td>
+<td>Fast on CPU</td>
 </tr>
 </tbody>
 </table>

mcp/index.html

@@ -827,7 +827,7 @@
 <li>Request hits SSE endpoint → Bearer token validated</li>
 <li>Celery task <code>classify_mcp</code> enqueued to Redis</li>
 <li>Worker loads Echo-DSRN-98M (baked into Docker image, no HF download)</li>
-<li>Sub-millisecond CPU inference → result streamed via SSE</li>
+<li>Fast CPU inference → result streamed via SSE</li>
 </ol>
 <p>No papers saved. No database writes. Stateless.</p>
 

search/search_index.json

@@ -1 +1 @@
-{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"],"fields":{"title":{"boost":1000.0},"text":{"boost":1.0},"tags":{"boost":1000000.0}}},"docs":[{"location":"","title":"OpenAIRE-AI-Research-Evaluator","text":"<p>A multi-model LLM-as-Judge pipeline that builds annotation datasets to train Echo-DSRN intent classifiers \u2014 entry for the OpenAIRE AI Hackathon 2026, co-organised by OpenAIRE and Alien Intelligence.</p> <p>The app streams scientific paper metadata from the OpenAIRE Graph API, classifies research intent with a 98M-parameter Echo-DSRN model (sub-millisecond CPU inference), and enlists multiple LLMs as judges to build a golden consensus dataset.</p>"},{"location":"#features","title":"Features","text":"<ul> <li>OpenAIRE Stream \u2014 live publication metadata via the OpenAIRE Graph API</li> <li>Collab Hub \u2014 community annotations with LLM judge consensus</li> <li>Free Text \u2014 ad-hoc classification of custom titles and abstracts</li> <li>Saved History \u2014 local browser storage with optional server-side sync</li> <li>Model Card \u2014 architecture specs, parameter census, and consumption code</li> </ul>"},{"location":"#hackathon-deliverable","title":"Hackathon Deliverable","text":"<p>This project is the artifact submission for the OpenAIRE AI Hackathon 2026.</p> <p>The accompanying story explains the question, journey, insight, and what others can reuse.</p> <p>Source \u00b7 OpenAIRE Graph \u00b7 Echo-DSRN Model</p>"},{"location":"architecture/","title":"Architecture","text":"Component Description Frontend Vue 3 / Vite dashboard \u2014 tabs: Model Card, Collab Hub, OpenAIRE Stream, Free Text, Saved History Backend FastAPI \u2014 inference, arXiv/OpenAIRE proxy, LLM judge orchestration, OAuth Classifier Echo-DSRN-98M (<code>EchoForSequenceClassification</code>) \u2014 sub-millisecond CPU inference Annotation DB SQLite (<code>data/collaborative.db</code>) \u2014 user classifications and LLM judge labels Worker Celery + Redis \u2014 async classification and LLM-as-Judge queue"},{"location":"architecture/#intent-labels","title":"Intent Labels","text":"Label Meaning <code>Methodology</code> Introduces a new method, model, or algorithm <code>Dataset</code> Introduces or documents a dataset or benchmark <code>Review</code> Surveys or synthesises existing work <code>Applied</code> Applies existing methods to a domain problem <code>Theoretical</code> Mathematical or formal analysis without empirical evaluation"},{"location":"architecture/#data-flow","title":"Data Flow","text":"<ol> <li>OpenAIRE Graph API streams paper metadata (title, abstract, DOI, language)</li> <li>Echo-DSRN-98M classifies research intent in &lt;1ms on CPU</li> <li>Results displayed in the Vue 3 dashboard with probability bars</li> <li>Optional: LLM-as-Judge enlists multiple LLMs to validate Echo's predictions</li> <li>Annotations stored in SQLite, forming a golden consensus dataset</li> </ol>"},{"location":"deployment/","title":"Kubernetes Deployment (k3s)","text":"<p>Deploy the OpenAIRE-AI-Research-Evaluator with Helm on k3s.</p>"},{"location":"deployment/#prerequisites","title":"Prerequisites","text":"<ul> <li>k3s cluster running</li> <li><code>helm</code> installed</li> <li>Access to the Docker image <code>ethicalabs/openaire-research-evaluator:latest</code></li> <li>Request access to the gated Echo-DSRN model</li> </ul>"},{"location":"deployment/#1-generate-postgresql-dump","title":"1. Generate PostgreSQL dump","text":"<p>The SQLite golden database must be migrated to PostgreSQL:</p> <pre><code>./scripts/sqlite_to_postgres.sh\n</code></pre> <p>Generates <code>data/collaborative_pg.sql</code> \u2014 ready for PostgreSQL import.</p>"},{"location":"deployment/#2-install","title":"2. Install","text":"<pre><code>helm install evaluator ./helm/openaire-research-evaluator \\\n  --set secrets.hfToken=hf_... \\\n  --set secrets.secretKey=$(openssl rand -hex 32) \\\n  --set ingress.host=openaire-eval.local \\\n  --set config.llmBaseUrl=http://192.168.1.66:13305/v1\n</code></pre> <p>Or create a <code>values.local.yaml</code>:</p> <pre><code>secrets:\n  hfToken: hf_...\n  secretKey: abc123...\n  hfClientId: ...\n  hfClientSecret: ...\ningress:\n  host: openaire-eval.local\nconfig:\n  llmBaseUrl: http://192.168.1.66:13305/v1\n</code></pre> <pre><code>helm install evaluator ./helm/openaire-research-evaluator -f values.local.yaml\n</code></pre>"},{"location":"deployment/#3-load-golden-data","title":"3. Load golden data","text":"<p>Wait for PostgreSQL to be ready, then import:</p> <pre><code>kubectl cp data/collaborative_pg.sql \\\n  $(kubectl get pod -l component=postgres -o jsonpath='{.items[0].metadata.name}'):/tmp/collaborative_pg.sql\n\nkubectl exec -it deployment/evaluator-openaire-research-evaluator-postgres -- \\\n  psql -U postgres -d echo_dsrn -f /tmp/collaborative_pg.sql\n\n# Reset auto-increment sequences after import (IDs were explicit in the dump)\nkubectl exec deployment/evaluator-openaire-research-evaluator-postgres -- \\\n  psql -U postgres -d echo_dsrn -c \\\n  \"SELECT setval('annotations_id_seq', COALESCE((SELECT MAX(id) FROM annotations), 1))\"\n</code></pre>"},{"location":"deployment/#4-access","title":"4. Access","text":"<pre><code>kubectl port-forward svc/evaluator-openaire-research-evaluator-web 7860:7860\n</code></pre> <p>App at <code>http://localhost:7860</code>. With Ingress configured, at <code>http://openaire-eval.local</code>.</p> <p>Local DNS setup: add to <code>/etc/hosts</code>:</p> <pre><code>echo \"127.0.0.1 openaire-eval.local\" | sudo tee -a /etc/hosts\n</code></pre> <p>Then access at <code>http://openaire-eval.local</code> (k3s Traefik listens on port 80).</p>"},{"location":"deployment/#components","title":"Components","text":"Component Service Port Web (FastAPI + Vue) <code>&lt;release&gt;-web</code> 7860 PostgreSQL <code>&lt;release&gt;-postgres</code> 5432 Redis <code>&lt;release&gt;-redis</code> 6379 Worker (Celery) \u2014 (internal) \u2014"},{"location":"deployment/#troubleshooting","title":"Troubleshooting","text":"<p>Check web pod logs for alembic migration status:</p> <pre><code>kubectl logs deployment/evaluator-openaire-research-evaluator-web\n</code></pre>"},{"location":"inference/","title":"Inference","text":"<p>Model: <code>ethicalabs/Echo-DSRN-v0.1.3-Research-Intent-CLF</code> \u2014 gated repo, request access before use.</p>"},{"location":"inference/#install","title":"Install","text":"<pre><code>pip install git+https://github.com/ethicalabs-ai/Echo-DSRN.git\n</code></pre>"},{"location":"inference/#classify","title":"Classify","text":"<pre><code>from echo_dsrn import EchoForSequenceClassification\nfrom transformers import AutoTokenizer\n\nmodel     = EchoForSequenceClassification.from_pretrained(\n    \"ethicalabs/Echo-DSRN-v0.1.3-Research-Intent-CLF\", trust_remote_code=True\n)\ntokenizer = AutoTokenizer.from_pretrained(\n    \"ethicalabs/Echo-DSRN-v0.1.3-Research-Intent-CLF\", trust_remote_code=True\n)\n\nlabel, probs = model.classify(\n    \"Title: Attention Is All You Need\\n\"\n    \"Abstract: We propose a new simple network architecture, the Transformer...\",\n    tokenizer=tokenizer,\n)\nprint(label, probs)\n# \u2192 Methodology  tensor([0.87, 0.03, 0.02, 0.06, 0.02])\n</code></pre>"},{"location":"inference/#model-specs","title":"Model Specs","text":"Property Value Architecture Echo-DSRN (Recurrent Neural Network) Parameters 98,266,629 (~98M) Layers 8 DSRN blocks Hidden dim 512 Attention heads 4 Vocab size 32,017 tokens Precision bfloat16 Inference Sub-millisecond on CPU <p>Run <code>curl /api/model/card</code> for live specs from the loaded model.</p>"},{"location":"license/","title":"License","text":""},{"location":"license/#code","title":"Code","text":"<p>Apache License 2.0 \u2014 see LICENSE.</p> <p>Copyright \u00a9 2026 ethicalabs.ai</p>"},{"location":"license/#openaire-graph","title":"OpenAIRE Graph","text":"<p>This project uses the OpenAIRE Graph API. All materials created by OpenAIRE are licensed under CC BY 4.0.</p>"},{"location":"license/#hackathon-deliverable","title":"Hackathon Deliverable","text":"<p>This project is submitted to the OpenAIRE AI Hackathon 2026. Per hackathon requirements, all submitted materials are available under a CC-BY license.</p>"},{"location":"license/#attribution","title":"Attribution","text":"<p>If you reuse this work, please cite:</p> <p>OpenAIRE-AI-Research-Evaluator \u2014 Multi-model LLM-as-Judge pipeline for research intent classification. ethicalabs.ai, 2026. Apache-2.0 / CC-BY 4.0. github.com/ethicalabs-ai/OpenAIRE-AI-Research-Evaluator</p>"},{"location":"llm-as-judge/","title":"LLM-as-Judge","text":"<p>The LLM-as-Judge pipeline evaluates Echo-DSRN predictions against high-capability LLMs and produces a golden consensus dataset in <code>data/collaborative.db</code>.</p>"},{"location":"llm-as-judge/#search-queries","title":"Search Queries","text":"<p>arXiv and OpenAIRE sources use keyword queries from <code>assets/topics.txt</code> \u2014 one query per line, blank lines and <code>#</code> comments ignored. Edit this file to target specific research domains.</p> <p>Container path: <code>/app/assets/topics.txt</code></p>"},{"location":"llm-as-judge/#setup","title":"Setup","text":"<p>Point at a lemonade or llama.cpp server via <code>.env</code>:</p> <pre><code>LLM_BASE_URL=http://192.168.1.40:13305/v1\n</code></pre> <p>Migrate the database:</p> <pre><code>uv run alembic -c backend/alembic.ini upgrade head\n</code></pre>"},{"location":"llm-as-judge/#llm-server-setup","title":"LLM Server Setup","text":"<p>Any OpenAI-compatible server works. Configuration via <code>.env</code>:</p> <pre><code>LLM_BASE_URL=&lt;server-url&gt;\nLLM_API_KEY=not-needed     # for local servers that skip auth\n</code></pre>"},{"location":"llm-as-judge/#ollama","title":"Ollama","text":"<pre><code>ollama pull qwen3:4b\nollama serve\n</code></pre> <pre><code>LLM_BASE_URL=http://localhost:11434/v1\n</code></pre>"},{"location":"llm-as-judge/#lm-studio","title":"LM Studio","text":"<p>Start the local server in the UI (Developer \u2192 Local Server). Default port:</p> <pre><code>LLM_BASE_URL=http://localhost:1234/v1\n</code></pre>"},{"location":"llm-as-judge/#lemonade-llamacpp","title":"Lemonade / llama.cpp","text":"<pre><code>lemonade --host 0.0.0.0 load --pinned Qwen3.6-35B-A3B-GGUF\n</code></pre> <pre><code>LLM_BASE_URL=http://192.168.1.40:13305/v1\n</code></pre>"},{"location":"llm-as-judge/#docker-llm-on-host","title":"Docker \u2014 LLM on host","text":"<p>When running the app in Docker and the LLM server on the host, use <code>host.docker.internal</code>:</p> <pre><code>LLM_BASE_URL=http://host.docker.internal:11434/v1\n</code></pre> <p>The <code>docker-compose.yaml</code> already includes <code>extra_hosts: host.docker.internal:host-gateway</code>.</p>"},{"location":"llm-as-judge/#batch-runner-sync","title":"Batch Runner \u2014 Sync","text":"<p>Loads models on lemonade, judges papers inline:</p> <pre><code>./scripts/run_judges.sh 100\n</code></pre> <p>Cycles through 11 GGUF models (Qwen3.6, Nemotron, Gemma, GPT-OSS, DeepSeek, GLM) against the catalog source.</p>"},{"location":"llm-as-judge/#batch-runner-async","title":"Batch Runner \u2014 Async","text":"<p>Dispatches Celery tasks to the worker, no model loading:</p> <pre><code>./scripts/run_judges.sh 100 --async\n</code></pre>"},{"location":"llm-as-judge/#kubernetes-k3s","title":"Kubernetes (k3s)","text":"<p>Run the judge CLI directly in the deployed web pod:</p> <pre><code>kubectl exec -it deployment/evaluator-openaire-research-evaluator-web -- \\\n  env LLM_BASE_URL=http://192.168.1.66:13305/v1 \\\n  python backend/judge_cli.py \\\n    --source catalog --n 100 \\\n    --model gpt-oss-20b-mxfp4-GGUF \\\n    --async\n</code></pre> <p>Tasks dispatch to the cluster's Celery worker, using the same Redis and PostgreSQL.</p>"},{"location":"llm-as-judge/#single-run","title":"Single Run","text":"<pre><code># Local GGUF via llama.cpp / lemonade\nuv run python backend/judge_cli.py \\\n    --source catalog --n 100 \\\n    --model Qwen3.6-35B-A3B-GGUF\n\n# Google Gemini (OpenAI-compatible endpoint)\nuv run python backend/judge_cli.py \\\n    --source catalog --n 100 \\\n    --model gemini-2.5-flash \\\n    --llm-url https://generativelanguage.googleapis.com/v1beta/openai/ \\\n    --llm-key ${GEMINI_API_KEY}\n</code></pre>"},{"location":"llm-as-judge/#pre-labeled-datasets","title":"Pre-labeled Datasets","text":"<p>Convert chat-format JSONL to flat records, then judge:</p> <pre><code>uv run python scripts/reformat_for_judge.py \\\n    ~/.ethicalabs/datasets/research-intent/train.jsonl \\\n    ~/.ethicalabs/datasets/research-intent/train_flat.jsonl\n\nuv run python backend/judge_cli.py \\\n    --source dataset \\\n    --dataset-path ~/.ethicalabs/datasets/research-intent/train_flat.jsonl \\\n    --n 100 --model Qwen3.6-35B-A3B-GGUF\n</code></pre>"},{"location":"llm-as-judge/#evaluation","title":"Evaluation","text":"<pre><code>uv run python backend/judge_eval.py                # all papers\nuv run python backend/judge_eval.py --paper-source dataset  # golden set only\n</code></pre> <p>Computes agreement rates, class-level accuracy, and per-model alignment stats.</p>"},{"location":"mcp/","title":"MCP \u2014 Agent Classification Endpoint","text":"<p>Expose Echo-DSRN-98M research intent classification to AI agents via the Model Context Protocol (SSE transport).</p>"},{"location":"mcp/#server-configuration","title":"Server Configuration","text":"<p>Add to your MCP client config (Claude Desktop, Cursor, Continue, etc.):</p> <pre><code>{\n  \"mcpServers\": {\n    \"echo-dsrn\": {\n      \"url\": \"http://localhost:7860/api/mcp/classify\",\n      \"transport\": \"sse\",\n      \"headers\": {\n        \"Authorization\": \"Bearer echo-dsrn-mcp-change-me-in-production\"\n      }\n    }\n  }\n}\n</code></pre> <p>Set <code>MCP_API_KEY</code> in <code>.env</code> to match:</p> <pre><code>MCP_API_KEY=echo-dsrn-mcp-change-me-in-production\n</code></pre>"},{"location":"mcp/#authentication","title":"Authentication","text":"<p>Bearer token via <code>Authorization</code> header. No query-string keys.</p>"},{"location":"mcp/#tools-exposed","title":"Tools exposed","text":"Tool Description <code>classify_intent</code> Classify a research paper by title + abstract into one of 5 intents"},{"location":"mcp/#input","title":"Input","text":"<pre><code>{\n  \"title\": \"Attention Is All You Need\",\n  \"abstract\": \"We propose a new simple network architecture, the Transformer...\"\n}\n</code></pre>"},{"location":"mcp/#output","title":"Output","text":"<pre><code>{\n  \"label\": \"Methodology\",\n  \"probabilities\": {\n    \"Methodology\": 0.87,\n    \"Dataset\": 0.03,\n    \"Review\": 0.02,\n    \"Applied\": 0.06,\n    \"Theoretical\": 0.02\n  }\n}\n</code></pre>"},{"location":"mcp/#raw-sse-usage","title":"Raw SSE usage","text":"<pre><code>curl -N -H \"Authorization: Bearer echo-dsrn-mcp-change-me-in-production\" \\\n  \"http://localhost:7860/api/mcp/classify?title=Attention+Is+All+You+Need&amp;abstract=We+propose+the+Transformer\"\n</code></pre> <pre><code>import requests\n\nresp = requests.get(\n    \"http://localhost:7860/api/mcp/classify\",\n    params={\"title\": \"Attention Is All You Need\", \"abstract\": \"We propose...\"},\n    headers={\"Authorization\": \"Bearer echo-dsrn-mcp-change-me-in-production\"},\n    stream=True,\n)\nfor line in resp.iter_lines():\n    if line:\n        print(line.decode())\n</code></pre>"},{"location":"mcp/#architecture","title":"Architecture","text":"<ol> <li>Request hits SSE endpoint \u2192 Bearer token validated</li> <li>Celery task <code>classify_mcp</code> enqueued to Redis</li> <li>Worker loads Echo-DSRN-98M (baked into Docker image, no HF download)</li> <li>Sub-millisecond CPU inference \u2192 result streamed via SSE</li> </ol> <p>No papers saved. No database writes. Stateless.</p>"},{"location":"oauth/","title":"OAuth \u2014 Hugging Face Login","text":"<p>The Collab Hub uses Hugging Face OAuth for authentication. Users log in with their HF account to vote on annotations and save papers.</p>"},{"location":"oauth/#setup","title":"Setup","text":""},{"location":"oauth/#1-create-an-oauth-app","title":"1. Create an OAuth App","text":"<p>Go to Hugging Face OAuth Apps and create a new app:</p> <ul> <li>Name: <code>Echo-DSRN Collab Hub</code></li> <li>Homepage URL: <code>http://localhost:7860</code> (or your production URL)</li> <li>Redirect URL: <code>http://localhost:7860/api/auth/callback/hf</code></li> </ul>"},{"location":"oauth/#2-configure-env","title":"2. Configure .env","text":"<pre><code>APP_BASE_URL=http://localhost:7860\nHF_CLIENT_ID=&lt;your-client-id&gt;\nHF_CLIENT_SECRET=&lt;your-client-secret&gt;\n</code></pre>"},{"location":"oauth/#3-production","title":"3. Production","text":"<p>Replace <code>localhost:7860</code> with your production domain. Both the Homepage URL and Redirect URL must match.</p> <pre><code>APP_BASE_URL=https://echo-dsrn.example.com\n</code></pre> <p>Redirect URL: <code>https://echo-dsrn.example.com/api/auth/callback/hf</code></p>"},{"location":"oauth/#how-it-works","title":"How it works","text":"<ol> <li>User clicks \"Login with Hugging Face\" in the Collab Hub</li> <li>Redirected to HF authorization page</li> <li>After consent, redirected back to <code>/api/auth/callback/hf</code></li> <li>Backend exchanges code for token, fetches user profile</li> <li>JWT session cookie set \u2014 user is authenticated</li> </ol> <p>User data stored: HF username, name, email, avatar URL, and optionally institution (from profile).</p>"},{"location":"quickstart/","title":"Quick Start","text":""},{"location":"quickstart/#prerequisites","title":"Prerequisites","text":"<ul> <li>Python \u2265 3.12 (tested with 3.12.3)</li> <li>Node.js \u2265 22 (tested with 22.22.3)</li> <li>npm \u2265 10 (tested with 10.9.8)</li> <li>Docker (optional, for containerised deployment)</li> </ul>"},{"location":"quickstart/#docker-compose","title":"Docker Compose","text":"<p>The Echo-DSRN classifier is a gated HF model. Before building:</p> <ol> <li>Request access to the model repo</li> <li>Create an HF access token with read permissions</li> <li>Add it to <code>.env</code>: <code>HF_TOKEN=hf_...</code></li> </ol> <pre><code>make rebuild      # builds Docker image with the gated model\ndocker compose up -d\n</code></pre> <p>App serves on <code>http://localhost:7860</code>. Redis, the web server, and a Celery worker start automatically.</p>"},{"location":"quickstart/#python","title":"Python","text":""},{"location":"quickstart/#uv","title":"uv","text":"<pre><code>uv run python backend/server.py\n</code></pre>"},{"location":"quickstart/#pip","title":"pip","text":"<pre><code>pip install -r requirements.txt\npython backend/server.py\n</code></pre> <p>Build the frontend first: <code>cd frontend &amp;&amp; npm ci &amp;&amp; npm run build</code>. App serves on <code>http://localhost:7860</code>.</p>"},{"location":"quickstart/#makefile","title":"Makefile","text":"<pre><code>make help          # list all targets\nmake up            # docker compose up -d\nmake build-docker  # build the Docker image\nmake test          # run pytest\nmake lint          # pre-commit hooks\nmake migrate       # alembic upgrade head\nmake judge N=100   # run LLM-as-Judge (sync)\nmake judge-async N=100  # dispatch to Celery worker\n</code></pre>"},{"location":"story/","title":"Hackathon Story","text":""},{"location":"story/#the-question","title":"The Question","text":"<p>How can we turn the OpenAIRE Graph \u2014 a living index of millions of research outputs \u2014 into validated, high-quality training data for AI classifiers, at scale and with minimal human effort?</p> <p>The OpenAIRE Graph aggregates publication metadata across disciplines, languages, and repositories.</p> <p>It is an extraordinary resource for building AI tools that understand research, But raw metadata is noisy: titles are truncated, abstracts vary in quality, and classification labels are absent.</p> <p>To train a classifier that can reliably distinguish a methodology paper from a dataset paper, a review from an applied case study, you need labeled data \u2014 and labeling thousands of papers by hand is slow, expensive, and fragile.</p> <p>Our question was: can we automate this with a feedback loop that combines a fast, lightweight classifier with a panel of LLM judges, using the OpenAIRE Graph as both the data source and the proving ground?</p>"},{"location":"story/#the-journey","title":"The Journey","text":""},{"location":"story/#phase-1-the-classifier","title":"Phase 1 \u2014 The Classifier","text":"<p>We started with Echo-DSRN, a recurrent neural network developed built from scratch by us.</p> <p>Echo-DSRN is small (98M-114M parameters) and runs on CPU with sub-millisecond latency, making it suitable for real-time streaming applications where a GPU is unavailable or impractical.</p> <p>We fine-tuned it on a 5-class research intent taxonomy:</p> <ul> <li>Methodology \u2014 introduces a new method, model, or algorithm</li> <li>Dataset \u2014 introduces or documents a dataset or benchmark</li> <li>Review \u2014 surveys or synthesises existing work</li> <li>Applied \u2014 applies existing methods to a domain problem</li> <li>Theoretical \u2014 mathematical or formal analysis without empirical evaluation</li> </ul> <p>The initial training set was ~6,000 records curated from PubMed, Semantic Scholar, Papers With Code, arXiv, and the OpenAIRE Graph itself \u2014 silver-labeled with keyword heuristics and arXiv category priors.</p> <p>The classifier worked well, but we had no systematic way to measure how well, or to improve it.</p>"},{"location":"story/#phase-2-the-app","title":"Phase 2 \u2014 The App","text":"<p>We built a Vue 3 + FastAPI web application that streams paper metadata from the OpenAIRE Graph API in real time.</p> <p>Users see papers appear with Echo-DSRN's predicted intent and a probability distribution across all five classes.</p> <p>They can classify custom text, save results to local history, and browse a Model Card showing the exact architecture, parameter census, and consumption code.</p> <p>But passive classification is only half the story.</p> <p>To improve the model, we needed ground truth \u2014 and that meant building a pipeline for annotation at scale.</p>"},{"location":"story/#phase-3-llm-as-judge","title":"Phase 3 \u2014 LLM-as-Judge","text":"<p>We designed an LLM-as-Judge pipeline where multiple high-capability language models evaluate Echo-DSRN's predictions and produce structured verdicts:</p> <ul> <li>Is the predicted label correct?</li> <li>If not, what should it be?</li> <li>Should the paper be flagged (garbled text, non-English, math-only)?</li> <li>What is the judge's confidence?</li> </ul> <p>Each judge receives the paper title, abstract, and Echo's prediction, then returns a JSON verdict following a strict Pydantic schema enforced via OpenAI's <code>response_format</code>.</p> <p>We support any OpenAI-compatible inference server: llama.cpp, vLLM, LM Studio, Google Gemini, and lemonade for local GGUF model orchestration.</p> <p>We assembled a panel of 11 models spanning architectures and parameter counts: <code>Qwen3.6-35B</code>, <code>Qwen3.6-27B</code>, <code>Qwen3.5-35B</code>, <code>Nemotron-3-Nano-30B</code>, <code>Gemma-4-26B</code>, <code>GPT-OSS-20B</code>, <code>GPT-OSS-120B</code>, <code>DeepSeek-Qwen3-8B</code>, <code>GLM-4.7-Flash</code>, <code>Qwen3.6-35B-A3B</code>, and <code>Qwen3.5-4B</code>.</p>"},{"location":"story/#phase-4-scale-with-celery","title":"Phase 4 \u2014 Scale with Celery","text":"<p>Running 11 models across thousands of papers synchronously is slow \u2014 each paper needs an LLM API call with its own latency.</p> <p>We added a Celery + Redis worker queue: the CLI dispatches fine-grained per-paper tasks, and workers pull from the queue and process them asynchronously. The same pipeline can run with a single command:</p> <pre><code>./scripts/run_judges.sh 100 --async\n</code></pre> <p>This dispatches 100 papers per model (1,100 total tasks) to the worker pool. The worker handles classification, LLM judging, and database persistence in one atomic unit, with idempotency guarantees \u2014 no paper is judged twice by the same model.</p>"},{"location":"story/#phase-5-the-golden-dataset","title":"Phase 5 \u2014 The Golden Dataset","text":"<p>All annotations are stored in a SQLite database (<code>data/collaborative.db</code>). The <code>judge_eval.py</code> script computes:</p> <ul> <li>Agreement rates between judges</li> <li>Per-class accuracy against ground truth (when available)</li> <li>Per-model alignment stats against Echo-DSRN</li> </ul> <p>The result is a golden consensus dataset where each paper has labels from multiple LLMs alongside Echo-DSRN's original prediction. This dataset becomes the training data for the next iteration of the classifier \u2014 closing the feedback loop.</p>"},{"location":"story/#the-insight","title":"The Insight","text":"<p>A multi-model LLM-as-Judge pipeline, fed by the OpenAIRE Graph, produces higher-quality training data than any single annotator \u2014 human or machine \u2014 at a fraction of the cost.</p> <p>Three findings stand out:</p> <ol> <li> <p>Diversity matters. Different models disagree on edge cases (applied vs. methodology, dataset vs. methodology), and consensus across architectures produces more robust labels than the strongest single model alone.</p> </li> <li> <p>The OpenAIRE Graph is a live data flywheel. New papers appear daily. A streaming classifier backed by LLM judges can continuously expand the training set without manual curation.</p> </li> <li> <p>CPU-first is viable. Echo-DSRN-98M classifies in under a millisecond on CPU, making it deployable anywhere \u2014 from a cloud VM to a Raspberry Pi. The heavy lifting (LLM judging) is offloaded to a remote GPU server, but the classification itself is free.</p> </li> </ol>"},{"location":"story/#what-others-can-reuse","title":"What Others Can Reuse","text":"<p>This project is designed to be remixed. Here is what you can take and adapt:</p> Component Reuse scenario LLM-as-Judge pipeline Swap the 5-class taxonomy for your own labels. Point at any OpenAI-compatible server. The judge schema, prompt, and evaluation scripts are label-agnostic. OpenAIRE Graph integration The keyword streaming + multilingual query approach works for any research domain. The fetcher handles both arXiv and OpenAIRE APIs. Async Celery worker pattern Fine-grained per-paper tasks with lazy database imports and idempotency \u2014 copy the pattern for any batch annotation workflow. Dataset preparation scripts <code>prepare_research_intent.py</code> and <code>expand_multilingual_openaire.py</code> build balanced, multilingual training sets from public APIs. Adapt the keyword banks and label taxonomy. Echo-DSRN classifier The model is on Hugging Face (gated \u2014 request access). <code>model.classify(title + abstract, tokenizer)</code> is the entire API. Swap the classification head for your own labels."},{"location":"story/#license","title":"License","text":"<p>Code: Apache 2.0. OpenAIRE Graph: CC BY 4.0. All hackathon materials: CC-BY.</p>"}]}
+{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"],"fields":{"title":{"boost":1000.0},"text":{"boost":1.0},"tags":{"boost":1000000.0}}},"docs":[{"location":"","title":"OpenAIRE-AI-Research-Evaluator","text":"<p>A multi-model LLM-as-Judge pipeline that builds annotation datasets to train Echo-DSRN intent classifiers \u2014 entry for the OpenAIRE AI Hackathon 2026, co-organised by OpenAIRE and Alien Intelligence.</p> <p>The app streams scientific paper metadata from the OpenAIRE Graph API, classifies research intent with a 98M-parameter Echo-DSRN model (fast CPU inference), and enlists multiple LLMs as judges to build a golden consensus dataset.</p>"},{"location":"#features","title":"Features","text":"<ul> <li>OpenAIRE Stream \u2014 live publication metadata via the OpenAIRE Graph API</li> <li>Collab Hub \u2014 community annotations with LLM judge consensus</li> <li>Free Text \u2014 ad-hoc classification of custom titles and abstracts</li> <li>Saved History \u2014 local browser storage with optional server-side sync</li> <li>Model Card \u2014 architecture specs, parameter census, and consumption code</li> </ul>"},{"location":"#hackathon-deliverable","title":"Hackathon Deliverable","text":"<p>This project is the artifact submission for the OpenAIRE AI Hackathon 2026.</p> <p>The accompanying story explains the question, journey, insight, and what others can reuse.</p> <p>Source \u00b7 OpenAIRE Graph \u00b7 Echo-DSRN Model</p>"},{"location":"architecture/","title":"Architecture","text":"Component Description Frontend Vue 3 / Vite dashboard \u2014 tabs: Model Card, Collab Hub, OpenAIRE Stream, Free Text, Saved History Backend FastAPI \u2014 inference, arXiv/OpenAIRE proxy, LLM judge orchestration, OAuth Classifier Echo-DSRN-98M (<code>EchoForSequenceClassification</code>) \u2014 fast CPU inference Annotation DB SQLite (<code>data/collaborative.db</code>) \u2014 user classifications and LLM judge labels Worker Celery + Redis \u2014 async classification and LLM-as-Judge queue"},{"location":"architecture/#intent-labels","title":"Intent Labels","text":"Label Meaning <code>Methodology</code> Introduces a new method, model, or algorithm <code>Dataset</code> Introduces or documents a dataset or benchmark <code>Review</code> Surveys or synthesises existing work <code>Applied</code> Applies existing methods to a domain problem <code>Theoretical</code> Mathematical or formal analysis without empirical evaluation"},{"location":"architecture/#data-flow","title":"Data Flow","text":"<ol> <li>OpenAIRE Graph API streams paper metadata (title, abstract, DOI, language)</li> <li>Echo-DSRN-98M classifies research intent in &lt;1ms on CPU</li> <li>Results displayed in the Vue 3 dashboard with probability bars</li> <li>Optional: LLM-as-Judge enlists multiple LLMs to validate Echo's predictions</li> <li>Annotations stored in SQLite, forming a golden consensus dataset</li> </ol>"},{"location":"deployment/","title":"Kubernetes Deployment (k3s)","text":"<p>Deploy the OpenAIRE-AI-Research-Evaluator with Helm on k3s.</p>"},{"location":"deployment/#prerequisites","title":"Prerequisites","text":"<ul> <li>k3s cluster running</li> <li><code>helm</code> installed</li> <li>Access to the Docker image <code>ethicalabs/openaire-research-evaluator:latest</code></li> <li>Request access to the gated Echo-DSRN model</li> </ul>"},{"location":"deployment/#1-generate-postgresql-dump","title":"1. Generate PostgreSQL dump","text":"<p>The SQLite golden database must be migrated to PostgreSQL:</p> <pre><code>./scripts/sqlite_to_postgres.sh\n</code></pre> <p>Generates <code>data/collaborative_pg.sql</code> \u2014 ready for PostgreSQL import.</p>"},{"location":"deployment/#2-install","title":"2. Install","text":"<pre><code>helm install evaluator ./helm/openaire-research-evaluator \\\n  --set secrets.hfToken=hf_... \\\n  --set secrets.secretKey=$(openssl rand -hex 32) \\\n  --set ingress.host=openaire-eval.local \\\n  --set config.llmBaseUrl=http://192.168.1.66:13305/v1\n</code></pre> <p>Or create a <code>values.local.yaml</code>:</p> <pre><code>secrets:\n  hfToken: hf_...\n  secretKey: abc123...\n  hfClientId: ...\n  hfClientSecret: ...\ningress:\n  host: openaire-eval.local\nconfig:\n  llmBaseUrl: http://192.168.1.66:13305/v1\n</code></pre> <pre><code>helm install evaluator ./helm/openaire-research-evaluator -f values.local.yaml\n</code></pre>"},{"location":"deployment/#3-load-golden-data","title":"3. Load golden data","text":"<p>Wait for PostgreSQL to be ready, then import:</p> <pre><code>kubectl cp data/collaborative_pg.sql \\\n  $(kubectl get pod -l component=postgres -o jsonpath='{.items[0].metadata.name}'):/tmp/collaborative_pg.sql\n\nkubectl exec -it deployment/evaluator-openaire-research-evaluator-postgres -- \\\n  psql -U postgres -d echo_dsrn -f /tmp/collaborative_pg.sql\n\n# Reset auto-increment sequences after import (IDs were explicit in the dump)\nkubectl exec deployment/evaluator-openaire-research-evaluator-postgres -- \\\n  psql -U postgres -d echo_dsrn -c \\\n  \"SELECT setval('annotations_id_seq', COALESCE((SELECT MAX(id) FROM annotations), 1))\"\n</code></pre>"},{"location":"deployment/#4-access","title":"4. Access","text":"<pre><code>kubectl port-forward svc/evaluator-openaire-research-evaluator-web 7860:7860\n</code></pre> <p>App at <code>http://localhost:7860</code>. With Ingress configured, at <code>http://openaire-eval.local</code>.</p> <p>Local DNS setup: add to <code>/etc/hosts</code>:</p> <pre><code>echo \"127.0.0.1 openaire-eval.local\" | sudo tee -a /etc/hosts\n</code></pre> <p>Then access at <code>http://openaire-eval.local</code> (k3s Traefik listens on port 80).</p>"},{"location":"deployment/#components","title":"Components","text":"Component Service Port Web (FastAPI + Vue) <code>&lt;release&gt;-web</code> 7860 PostgreSQL <code>&lt;release&gt;-postgres</code> 5432 Redis <code>&lt;release&gt;-redis</code> 6379 Worker (Celery) \u2014 (internal) \u2014"},{"location":"deployment/#troubleshooting","title":"Troubleshooting","text":"<p>Check web pod logs for alembic migration status:</p> <pre><code>kubectl logs deployment/evaluator-openaire-research-evaluator-web\n</code></pre>"},{"location":"inference/","title":"Inference","text":"<p>Model: <code>ethicalabs/Echo-DSRN-v0.1.3-Research-Intent-CLF</code> \u2014 gated repo, request access before use.</p>"},{"location":"inference/#install","title":"Install","text":"<pre><code>pip install git+https://github.com/ethicalabs-ai/Echo-DSRN.git\n</code></pre>"},{"location":"inference/#classify","title":"Classify","text":"<pre><code>from echo_dsrn import EchoForSequenceClassification\nfrom transformers import AutoTokenizer\n\nmodel     = EchoForSequenceClassification.from_pretrained(\n    \"ethicalabs/Echo-DSRN-v0.1.3-Research-Intent-CLF\", trust_remote_code=True\n)\ntokenizer = AutoTokenizer.from_pretrained(\n    \"ethicalabs/Echo-DSRN-v0.1.3-Research-Intent-CLF\", trust_remote_code=True\n)\n\nlabel, probs = model.classify(\n    \"Title: Attention Is All You Need\\n\"\n    \"Abstract: We propose a new simple network architecture, the Transformer...\",\n    tokenizer=tokenizer,\n)\nprint(label, probs)\n# \u2192 Methodology  tensor([0.87, 0.03, 0.02, 0.06, 0.02])\n</code></pre>"},{"location":"inference/#model-specs","title":"Model Specs","text":"Property Value Architecture Echo-DSRN (Recurrent Neural Network) Parameters 98,266,629 (~98M) Layers 8 DSRN blocks Hidden dim 512 Attention heads 4 Vocab size 32,017 tokens Precision bfloat16 Inference Fast on CPU <p>Run <code>curl /api/model/card</code> for live specs from the loaded model.</p>"},{"location":"license/","title":"License","text":""},{"location":"license/#code","title":"Code","text":"<p>Apache License 2.0 \u2014 see LICENSE.</p> <p>Copyright \u00a9 2026 ethicalabs.ai</p>"},{"location":"license/#openaire-graph","title":"OpenAIRE Graph","text":"<p>This project uses the OpenAIRE Graph API. All materials created by OpenAIRE are licensed under CC BY 4.0.</p>"},{"location":"license/#hackathon-deliverable","title":"Hackathon Deliverable","text":"<p>This project is submitted to the OpenAIRE AI Hackathon 2026. Per hackathon requirements, all submitted materials are available under a CC-BY license.</p>"},{"location":"license/#attribution","title":"Attribution","text":"<p>If you reuse this work, please cite:</p> <p>OpenAIRE-AI-Research-Evaluator \u2014 Multi-model LLM-as-Judge pipeline for research intent classification. ethicalabs.ai, 2026. Apache-2.0 / CC-BY 4.0. github.com/ethicalabs-ai/OpenAIRE-AI-Research-Evaluator</p>"},{"location":"llm-as-judge/","title":"LLM-as-Judge","text":"<p>The LLM-as-Judge pipeline evaluates Echo-DSRN predictions against high-capability LLMs and produces a golden consensus dataset in <code>data/collaborative.db</code>.</p>"},{"location":"llm-as-judge/#search-queries","title":"Search Queries","text":"<p>arXiv and OpenAIRE sources use keyword queries from <code>assets/topics.txt</code> \u2014 one query per line, blank lines and <code>#</code> comments ignored. Edit this file to target specific research domains.</p> <p>Container path: <code>/app/assets/topics.txt</code></p>"},{"location":"llm-as-judge/#setup","title":"Setup","text":"<p>Point at a lemonade or llama.cpp server via <code>.env</code>:</p> <pre><code>LLM_BASE_URL=http://192.168.1.40:13305/v1\n</code></pre> <p>Migrate the database:</p> <pre><code>uv run alembic -c backend/alembic.ini upgrade head\n</code></pre>"},{"location":"llm-as-judge/#llm-server-setup","title":"LLM Server Setup","text":"<p>Any OpenAI-compatible server works. Configuration via <code>.env</code>:</p> <pre><code>LLM_BASE_URL=&lt;server-url&gt;\nLLM_API_KEY=not-needed     # for local servers that skip auth\n</code></pre>"},{"location":"llm-as-judge/#ollama","title":"Ollama","text":"<pre><code>ollama pull qwen3:4b\nollama serve\n</code></pre> <pre><code>LLM_BASE_URL=http://localhost:11434/v1\n</code></pre>"},{"location":"llm-as-judge/#lm-studio","title":"LM Studio","text":"<p>Start the local server in the UI (Developer \u2192 Local Server). Default port:</p> <pre><code>LLM_BASE_URL=http://localhost:1234/v1\n</code></pre>"},{"location":"llm-as-judge/#lemonade-llamacpp","title":"Lemonade / llama.cpp","text":"<pre><code>lemonade --host 0.0.0.0 load --pinned Qwen3.6-35B-A3B-GGUF\n</code></pre> <pre><code>LLM_BASE_URL=http://192.168.1.40:13305/v1\n</code></pre>"},{"location":"llm-as-judge/#docker-llm-on-host","title":"Docker \u2014 LLM on host","text":"<p>When running the app in Docker and the LLM server on the host, use <code>host.docker.internal</code>:</p> <pre><code>LLM_BASE_URL=http://host.docker.internal:11434/v1\n</code></pre> <p>The <code>docker-compose.yaml</code> already includes <code>extra_hosts: host.docker.internal:host-gateway</code>.</p>"},{"location":"llm-as-judge/#batch-runner-sync","title":"Batch Runner \u2014 Sync","text":"<p>Loads models on lemonade, judges papers inline:</p> <pre><code>./scripts/run_judges.sh 100\n</code></pre> <p>Cycles through 11 GGUF models (Qwen3.6, Nemotron, Gemma, GPT-OSS, DeepSeek, GLM) against the catalog source.</p>"},{"location":"llm-as-judge/#batch-runner-async","title":"Batch Runner \u2014 Async","text":"<p>Dispatches Celery tasks to the worker, no model loading:</p> <pre><code>./scripts/run_judges.sh 100 --async\n</code></pre>"},{"location":"llm-as-judge/#kubernetes-k3s","title":"Kubernetes (k3s)","text":"<p>Run the judge CLI directly in the deployed web pod:</p> <pre><code>kubectl exec -it deployment/evaluator-openaire-research-evaluator-web -- \\\n  env LLM_BASE_URL=http://192.168.1.66:13305/v1 \\\n  python backend/judge_cli.py \\\n    --source catalog --n 100 \\\n    --model gpt-oss-20b-mxfp4-GGUF \\\n    --async\n</code></pre> <p>Tasks dispatch to the cluster's Celery worker, using the same Redis and PostgreSQL.</p>"},{"location":"llm-as-judge/#single-run","title":"Single Run","text":"<pre><code># Local GGUF via llama.cpp / lemonade\nuv run python backend/judge_cli.py \\\n    --source catalog --n 100 \\\n    --model Qwen3.6-35B-A3B-GGUF\n\n# Google Gemini (OpenAI-compatible endpoint)\nuv run python backend/judge_cli.py \\\n    --source catalog --n 100 \\\n    --model gemini-2.5-flash \\\n    --llm-url https://generativelanguage.googleapis.com/v1beta/openai/ \\\n    --llm-key ${GEMINI_API_KEY}\n</code></pre>"},{"location":"llm-as-judge/#pre-labeled-datasets","title":"Pre-labeled Datasets","text":"<p>Convert chat-format JSONL to flat records, then judge:</p> <pre><code>uv run python scripts/reformat_for_judge.py \\\n    ~/.ethicalabs/datasets/research-intent/train.jsonl \\\n    ~/.ethicalabs/datasets/research-intent/train_flat.jsonl\n\nuv run python backend/judge_cli.py \\\n    --source dataset \\\n    --dataset-path ~/.ethicalabs/datasets/research-intent/train_flat.jsonl \\\n    --n 100 --model Qwen3.6-35B-A3B-GGUF\n</code></pre>"},{"location":"llm-as-judge/#evaluation","title":"Evaluation","text":"<pre><code>uv run python backend/judge_eval.py                # all papers\nuv run python backend/judge_eval.py --paper-source dataset  # golden set only\n</code></pre> <p>Computes agreement rates, class-level accuracy, and per-model alignment stats.</p>"},{"location":"mcp/","title":"MCP \u2014 Agent Classification Endpoint","text":"<p>Expose Echo-DSRN-98M research intent classification to AI agents via the Model Context Protocol (SSE transport).</p>"},{"location":"mcp/#server-configuration","title":"Server Configuration","text":"<p>Add to your MCP client config (Claude Desktop, Cursor, Continue, etc.):</p> <pre><code>{\n  \"mcpServers\": {\n    \"echo-dsrn\": {\n      \"url\": \"http://localhost:7860/api/mcp/classify\",\n      \"transport\": \"sse\",\n      \"headers\": {\n        \"Authorization\": \"Bearer echo-dsrn-mcp-change-me-in-production\"\n      }\n    }\n  }\n}\n</code></pre> <p>Set <code>MCP_API_KEY</code> in <code>.env</code> to match:</p> <pre><code>MCP_API_KEY=echo-dsrn-mcp-change-me-in-production\n</code></pre>"},{"location":"mcp/#authentication","title":"Authentication","text":"<p>Bearer token via <code>Authorization</code> header. No query-string keys.</p>"},{"location":"mcp/#tools-exposed","title":"Tools exposed","text":"Tool Description <code>classify_intent</code> Classify a research paper by title + abstract into one of 5 intents"},{"location":"mcp/#input","title":"Input","text":"<pre><code>{\n  \"title\": \"Attention Is All You Need\",\n  \"abstract\": \"We propose a new simple network architecture, the Transformer...\"\n}\n</code></pre>"},{"location":"mcp/#output","title":"Output","text":"<pre><code>{\n  \"label\": \"Methodology\",\n  \"probabilities\": {\n    \"Methodology\": 0.87,\n    \"Dataset\": 0.03,\n    \"Review\": 0.02,\n    \"Applied\": 0.06,\n    \"Theoretical\": 0.02\n  }\n}\n</code></pre>"},{"location":"mcp/#raw-sse-usage","title":"Raw SSE usage","text":"<pre><code>curl -N -H \"Authorization: Bearer echo-dsrn-mcp-change-me-in-production\" \\\n  \"http://localhost:7860/api/mcp/classify?title=Attention+Is+All+You+Need&amp;abstract=We+propose+the+Transformer\"\n</code></pre> <pre><code>import requests\n\nresp = requests.get(\n    \"http://localhost:7860/api/mcp/classify\",\n    params={\"title\": \"Attention Is All You Need\", \"abstract\": \"We propose...\"},\n    headers={\"Authorization\": \"Bearer echo-dsrn-mcp-change-me-in-production\"},\n    stream=True,\n)\nfor line in resp.iter_lines():\n    if line:\n        print(line.decode())\n</code></pre>"},{"location":"mcp/#architecture","title":"Architecture","text":"<ol> <li>Request hits SSE endpoint \u2192 Bearer token validated</li> <li>Celery task <code>classify_mcp</code> enqueued to Redis</li> <li>Worker loads Echo-DSRN-98M (baked into Docker image, no HF download)</li> <li>Fast CPU inference \u2192 result streamed via SSE</li> </ol> <p>No papers saved. No database writes. Stateless.</p>"},{"location":"oauth/","title":"OAuth \u2014 Hugging Face Login","text":"<p>The Collab Hub uses Hugging Face OAuth for authentication. Users log in with their HF account to vote on annotations and save papers.</p>"},{"location":"oauth/#setup","title":"Setup","text":""},{"location":"oauth/#1-create-an-oauth-app","title":"1. Create an OAuth App","text":"<p>Go to Hugging Face OAuth Apps and create a new app:</p> <ul> <li>Name: <code>Echo-DSRN Collab Hub</code></li> <li>Homepage URL: <code>http://localhost:7860</code> (or your production URL)</li> <li>Redirect URL: <code>http://localhost:7860/api/auth/callback/hf</code></li> </ul>"},{"location":"oauth/#2-configure-env","title":"2. Configure .env","text":"<pre><code>APP_BASE_URL=http://localhost:7860\nHF_CLIENT_ID=&lt;your-client-id&gt;\nHF_CLIENT_SECRET=&lt;your-client-secret&gt;\n</code></pre>"},{"location":"oauth/#3-production","title":"3. Production","text":"<p>Replace <code>localhost:7860</code> with your production domain. Both the Homepage URL and Redirect URL must match.</p> <pre><code>APP_BASE_URL=https://echo-dsrn.example.com\n</code></pre> <p>Redirect URL: <code>https://echo-dsrn.example.com/api/auth/callback/hf</code></p>"},{"location":"oauth/#how-it-works","title":"How it works","text":"<ol> <li>User clicks \"Login with Hugging Face\" in the Collab Hub</li> <li>Redirected to HF authorization page</li> <li>After consent, redirected back to <code>/api/auth/callback/hf</code></li> <li>Backend exchanges code for token, fetches user profile</li> <li>JWT session cookie set \u2014 user is authenticated</li> </ol> <p>User data stored: HF username, name, email, avatar URL, and optionally institution (from profile).</p>"},{"location":"quickstart/","title":"Quick Start","text":""},{"location":"quickstart/#prerequisites","title":"Prerequisites","text":"<ul> <li>Python \u2265 3.12 (tested with 3.12.3)</li> <li>Node.js \u2265 22 (tested with 22.22.3)</li> <li>npm \u2265 10 (tested with 10.9.8)</li> <li>Docker (optional, for containerised deployment)</li> </ul>"},{"location":"quickstart/#docker-compose","title":"Docker Compose","text":"<p>The Echo-DSRN classifier is a gated HF model. Before building:</p> <ol> <li>Request access to the model repo</li> <li>Create an HF access token with read permissions</li> <li>Add it to <code>.env</code>: <code>HF_TOKEN=hf_...</code></li> </ol> <pre><code>make rebuild      # builds Docker image with the gated model\ndocker compose up -d\n</code></pre> <p>App serves on <code>http://localhost:7860</code>. Redis, the web server, and a Celery worker start automatically.</p>"},{"location":"quickstart/#python","title":"Python","text":""},{"location":"quickstart/#uv","title":"uv","text":"<pre><code>uv run python backend/server.py\n</code></pre>"},{"location":"quickstart/#pip","title":"pip","text":"<pre><code>pip install -r requirements.txt\npython backend/server.py\n</code></pre> <p>Build the frontend first: <code>cd frontend &amp;&amp; npm ci &amp;&amp; npm run build</code>. App serves on <code>http://localhost:7860</code>.</p>"},{"location":"quickstart/#makefile","title":"Makefile","text":"<pre><code>make help          # list all targets\nmake up            # docker compose up -d\nmake build-docker  # build the Docker image\nmake test          # run pytest\nmake lint          # pre-commit hooks\nmake migrate       # alembic upgrade head\nmake judge N=100   # run LLM-as-Judge (sync)\nmake judge-async N=100  # dispatch to Celery worker\n</code></pre>"},{"location":"story/","title":"Hackathon Story","text":""},{"location":"story/#the-question","title":"The Question","text":"<p>How can we turn the OpenAIRE Graph \u2014 a living index of millions of research outputs \u2014 into validated, high-quality training data for AI classifiers, at scale and with minimal human effort?</p> <p>The OpenAIRE Graph aggregates publication metadata across disciplines, languages, and repositories.</p> <p>It is an extraordinary resource for building AI tools that understand research, But raw metadata is noisy: titles are truncated, abstracts vary in quality, and classification labels are absent.</p> <p>To train a classifier that can reliably distinguish a methodology paper from a dataset paper, a review from an applied case study, you need labeled data \u2014 and labeling thousands of papers by hand is slow, expensive, and fragile.</p> <p>Our question was: can we automate this with a feedback loop that combines a fast, lightweight classifier with a panel of LLM judges, using the OpenAIRE Graph as both the data source and the proving ground?</p>"},{"location":"story/#the-journey","title":"The Journey","text":""},{"location":"story/#phase-1-the-classifier","title":"Phase 1 \u2014 The Classifier","text":"<p>We started with Echo-DSRN, a recurrent neural network developed built from scratch by us.</p> <p>Echo-DSRN is small (98M-114M parameters) and runs efficiently on CPU, making it suitable for real-time streaming applications where a GPU is unavailable or impractical.</p> <p>We fine-tuned it on a 5-class research intent taxonomy:</p> <ul> <li>Methodology \u2014 introduces a new method, model, or algorithm</li> <li>Dataset \u2014 introduces or documents a dataset or benchmark</li> <li>Review \u2014 surveys or synthesises existing work</li> <li>Applied \u2014 applies existing methods to a domain problem</li> <li>Theoretical \u2014 mathematical or formal analysis without empirical evaluation</li> </ul> <p>The initial training set was ~6,000 records curated from PubMed, Semantic Scholar, Papers With Code, arXiv, and the OpenAIRE Graph itself \u2014 silver-labeled with keyword heuristics and arXiv category priors.</p> <p>The classifier worked well, but we had no systematic way to measure how well, or to improve it.</p>"},{"location":"story/#phase-2-the-app","title":"Phase 2 \u2014 The App","text":"<p>We built a Vue 3 + FastAPI web application that streams paper metadata from the OpenAIRE Graph API in real time.</p> <p>Users see papers appear with Echo-DSRN's predicted intent and a probability distribution across all five classes.</p> <p>They can classify custom text, save results to local history, and browse a Model Card showing the exact architecture, parameter census, and consumption code.</p> <p>But passive classification is only half the story.</p> <p>To improve the model, we needed ground truth \u2014 and that meant building a pipeline for annotation at scale.</p>"},{"location":"story/#phase-3-llm-as-judge","title":"Phase 3 \u2014 LLM-as-Judge","text":"<p>We designed an LLM-as-Judge pipeline where multiple high-capability language models evaluate Echo-DSRN's predictions and produce structured verdicts:</p> <ul> <li>Is the predicted label correct?</li> <li>If not, what should it be?</li> <li>Should the paper be flagged (garbled text, non-English, math-only)?</li> <li>What is the judge's confidence?</li> </ul> <p>Each judge receives the paper title, abstract, and Echo's prediction, then returns a JSON verdict following a strict Pydantic schema enforced via OpenAI's <code>response_format</code>.</p> <p>We support any OpenAI-compatible inference server: llama.cpp, vLLM, LM Studio, Google Gemini, and lemonade for local GGUF model orchestration.</p> <p>We assembled a panel of 11 models spanning architectures and parameter counts: <code>Qwen3.6-35B</code>, <code>Qwen3.6-27B</code>, <code>Qwen3.5-35B</code>, <code>Nemotron-3-Nano-30B</code>, <code>Gemma-4-26B</code>, <code>GPT-OSS-20B</code>, <code>GPT-OSS-120B</code>, <code>DeepSeek-Qwen3-8B</code>, <code>GLM-4.7-Flash</code>, <code>Qwen3.6-35B-A3B</code>, and <code>Qwen3.5-4B</code>.</p>"},{"location":"story/#phase-4-scale-with-celery","title":"Phase 4 \u2014 Scale with Celery","text":"<p>Running 11 models across thousands of papers synchronously is slow \u2014 each paper needs an LLM API call with its own latency.</p> <p>We added a Celery + Redis worker queue: the CLI dispatches fine-grained per-paper tasks, and workers pull from the queue and process them asynchronously. The same pipeline can run with a single command:</p> <pre><code>./scripts/run_judges.sh 100 --async\n</code></pre> <p>This dispatches 100 papers per model (1,100 total tasks) to the worker pool. The worker handles classification, LLM judging, and database persistence in one atomic unit, with idempotency guarantees \u2014 no paper is judged twice by the same model.</p>"},{"location":"story/#phase-5-the-golden-dataset","title":"Phase 5 \u2014 The Golden Dataset","text":"<p>All annotations are stored in a SQLite database (<code>data/collaborative.db</code>). The <code>judge_eval.py</code> script computes:</p> <ul> <li>Agreement rates between judges</li> <li>Per-class accuracy against ground truth (when available)</li> <li>Per-model alignment stats against Echo-DSRN</li> </ul> <p>The result is a golden consensus dataset where each paper has labels from multiple LLMs alongside Echo-DSRN's original prediction. This dataset becomes the training data for the next iteration of the classifier \u2014 closing the feedback loop.</p>"},{"location":"story/#the-insight","title":"The Insight","text":"<p>A multi-model LLM-as-Judge pipeline, fed by the OpenAIRE Graph, produces higher-quality training data than any single annotator \u2014 human or machine \u2014 at a fraction of the cost.</p> <p>Three findings stand out:</p> <ol> <li> <p>Diversity matters. Different models disagree on edge cases (applied vs. methodology, dataset vs. methodology), and consensus across architectures produces more robust labels than the strongest single model alone.</p> </li> <li> <p>The OpenAIRE Graph is a live data flywheel. New papers appear daily. A streaming classifier backed by LLM judges can continuously expand the training set without manual curation.</p> </li> <li> <p>CPU-first is viable. Echo-DSRN-98M runs fast enough on CPU to be deployable anywhere \u2014 from a cloud VM to a Raspberry Pi. The heavy lifting (LLM judging) is offloaded to a remote GPU server, but the classification itself is free.</p> </li> </ol>"},{"location":"story/#what-others-can-reuse","title":"What Others Can Reuse","text":"<p>This project is designed to be remixed. Here is what you can take and adapt:</p> Component Reuse scenario LLM-as-Judge pipeline Swap the 5-class taxonomy for your own labels. Point at any OpenAI-compatible server. The judge schema, prompt, and evaluation scripts are label-agnostic. OpenAIRE Graph integration The keyword streaming + multilingual query approach works for any research domain. The fetcher handles both arXiv and OpenAIRE APIs. Async Celery worker pattern Fine-grained per-paper tasks with lazy database imports and idempotency \u2014 copy the pattern for any batch annotation workflow. Dataset preparation scripts <code>prepare_research_intent.py</code> and <code>expand_multilingual_openaire.py</code> build balanced, multilingual training sets from public APIs. Adapt the keyword banks and label taxonomy. Echo-DSRN classifier The model is on Hugging Face (gated \u2014 request access). <code>model.classify(title + abstract, tokenizer)</code> is the entire API. Swap the classification head for your own labels."},{"location":"story/#license","title":"License","text":"<p>Code: Apache 2.0. OpenAIRE Graph: CC BY 4.0. All hackathon materials: CC-BY.</p>"}]}

story/index.html

@@ -831,7 +831,7 @@
 <h2 id="the-journey">The Journey</h2>
 <h3 id="phase-1-the-classifier">Phase 1 — The Classifier</h3>
 <p>We started with <a href="https://www.ethicalabs.ai/research/echo-dsrn/"><strong>Echo-DSRN</strong></a>, a recurrent neural network developed built from scratch by us.</p>
-<p>Echo-DSRN is small (98M-114M parameters) and runs on CPU with sub-millisecond latency, making it suitable for real-time streaming applications where a GPU is unavailable or impractical.</p>
+<p>Echo-DSRN is small (98M-114M parameters) and runs efficiently on CPU, making it suitable for real-time streaming applications where a GPU is unavailable or impractical.</p>
 <p>We fine-tuned it on a 5-class research intent taxonomy:</p>
 <ul>
 <li><strong>Methodology</strong> — introduces a new method, model, or algorithm</li>
@@ -885,7 +885,7 @@
 <p><strong>The OpenAIRE Graph is a live data flywheel.</strong> New papers appear daily. A streaming classifier backed by LLM judges can continuously expand the training set without manual curation.</p>
 </li>
 <li>
-<p><strong>CPU-first is viable.</strong> Echo-DSRN-98M classifies in under a millisecond on CPU, making it deployable anywhere — from a cloud VM to a Raspberry Pi. The heavy lifting (LLM judging) is offloaded to a remote GPU server, but the classification itself is free.</p>
+<p><strong>CPU-first is viable.</strong> Echo-DSRN-98M runs fast enough on CPU to be deployable anywhere — from a cloud VM to a Raspberry Pi. The heavy lifting (LLM judging) is offloaded to a remote GPU server, but the classification itself is free.</p>
 </li>
 </ol>
 <hr />