On this page
About This Challenge
Welcome to the MecCog Challenge, a collaborative effort to map the mechanisms by which the APOE4 genetic variant increases the risk of Alzheimer’s disease.
What You’ll Do
Your task is to find relevant research papers and extract key experimental findings that relate to the specific APOE4-Alzheimer’s hypothesis you select.
For each hypothesis, you will:
Search the scientific literature (PubMed, preprints, databases, web sources)
Identify papers that bear on that mechanism
Extract experimental findings from those papers (measurements, effects, experimental conditions)
Record everything in a structured spreadsheet that we validate
This is Step 1 of a multi-phase challenge. Later phases will ask you to assess evidence, estimate mechanism confidence, and design experiments. For now we focus on finding and documenting what is already known.
The Hypotheses
Hypotheses are released in batches, starting November 9th, 2026. The current batch is listed on the hypotheses page.
Pick the hypothesis you want to work on. Do not catalog Alzheimer’s mechanisms in general. We need findings that directly test or inform the one hypothesis you picked, and a submission padded with general Alzheimer’s literature scores worse than a short, focused one.
Each hypothesis carries qualifying clauses, and they are part of it: a finding in aged or AD tissue does not test a hypothesis stated for non-aged, non-AD tissue, and should be recorded with that mismatch noted rather than presented as direct support.
Teams
There is no separate registration step. Your first submission is what registers your team. The Method sheet carries the team name, the team leader and the leader’s email, and those three cells are what tell us who produced the work and where to write.
Teams. Every submission belongs to a team. One person is named as the team leader, and a submission with no leader does not validate.
Working alone is fine. Give your team a name and put yourself down as its leader. The team exists so that submissions have an owner and a point of contact, not to require you to find collaborators.
Identity. Use an academic or registered company email address for the team leader where you have one. If you do not, use whatever address reaches you and say in your README who you are and what you do. We may write before the submission is accepted.
You may belong to more than one team, but creating a second team requires a written explanation, for example that your group has two genuinely different methods and wants both assessed. Creating several teams to multiply your submissions is not allowed.
Challenge Rules
What We Are Asking For
Search widely. PubMed, the preprint servers, Google Scholar, databases, and anything else you think bears on the hypothesis. We would rather see a source you end up setting aside than miss one that mattered.
Extract findings precisely. Each one needs:
A one-sentence description of what was done and what was observed
A direct quote from the paper, or N/A if none is available
The experimental system, described as precisely as the paper allows. Not just “astrocytes” but the source of the cells; not just “mouse” but the strain
Where in the paper the finding appears, at panel level. Fig 4C, not Fig 4
The statistics, with their provenance (see below)
The directness class, saying how the experiment lines up with the hypothesis, and the strength of that link
Keep the findings that argue against the hypothesis. This one matters more to us than any other. A result that contradicts the hypothesis, or that shows no effect, is evidence, and we need it as much as we need the supporting ones. Record it like any other finding, with the same care over the quote and the statistics. The whole reason this project exists is a literature in which most reported mechanisms are probably wrong; a submission that quietly drops the inconvenient results reproduces that problem inside the challenge. The cross-comparison between submissions tends to show when it has happened.
Leave no cell empty. Every cell carries either a value or the literal N/A. A blank tells us nothing, and we cannot tell it apart from an oversight.
Get the metadata right. DOI, PubMed ID where the source has one, and source type, for every paper. These are what let us line your submission up against everyone else’s.
Validate before you submit. Use the Validate button on the submission page, or run the validator yourself beforehand; the download link is on the shared resources page. A submission that fails validation cannot be uploaded, so it is worth running early rather than on the deadline.
Publish your method. Name a public repository or archive holding a README that describes how your submission was produced, and the commit it was built from. Both go on the Method sheet of the spreadsheet, described below. A list of findings without a method is a result without a protocol, which is why a submission without them does not validate.
Where Submissions Go Wrong
These are the five we expect to see, and what each one costs.
Padding the sheet. Include only the papers and findings that genuinely bear on your hypothesis. A submission swollen with general Alzheimer’s literature scores worse than a short, focused one. This is not a licence to drop a finding that argues against the hypothesis: relevance is about the subject, not the direction of the result. One submission carries at most 25 papers, and the validator enforces it, so order them by relevance from the start and keep the ones that earn their place.
Fabrication. Every finding must exist in the paper at the location you cite. Misattributions are caught, and they disqualify the submission.
An estimate presented as a reported value. You may read an effect size off a bar chart, or count the plotted points to get a sample size, and we would rather have the estimate than a blank. A P value is the one thing you may never estimate, for the reason given under Recording what the paper does not report. But an estimate has to say so: a leading ~ on the value, and the provenance code estimated_from_figure. An estimate recorded as though the paper printed it is a fabrication, and it is the kind we are most likely to catch, because other submissions will have read the same panel.
Invented identifiers. DOIs and PubMed IDs must correspond to real, published sources. A plausible-looking identifier that resolves to nothing is worse than N/A.
Working ahead. This is Step 1 only. Assessing confidence, designing experiments and building mechanism schemas all come later, and doing them now will not count in your favour.
Submission Format
One spreadsheet per hypothesis. Download the official template from the shared resources page.
Each submission is limited to 25 papers.
Use the hypothesis statement exactly as published, including its qualifying clauses, and copy it verbatim into cell A2 of the Submission sheet.
Structure
The template has two sheets that the validator reads, Submission and Method, already named. Do not rename them. A third sheet, How to fill this in, comes last and is there for you; the validator ignores it.
The Submission sheet:
Row 1: column labels, provided in the template. All of them must be present, in the template’s order
Row 2: the hypothesis in column A. The rest of row 2 stays empty
Row 3 onward: one block per paper, a paper row followed by one row per finding from that paper
The Method sheet holds your team, who leads it and how to reach them, and the link to your method, seven labelled rows described under Publishing your method.
The columns of the Submission sheet
| Column | Content | Accepted values |
|---|---|---|
| Hypothesis | The hypothesis you selected. Row 2 only | The statement copied verbatim from the hypotheses page. Empty on every other row |
| DOI | The source’s DOI | A DOI, or N/A for a source that has none |
| Paper/source type | What kind of source it is | Exactly one of PubMed published, PubMed preprint, Web article, Database, Other |
| PubMed ID | Required for any source catalogued in PubMed | Digits only, or N/A |
| Paper/Finding ID | What the row is: a paper or one of its findings | P1, P2 and so on for papers, in order of relevance, P1 holding the findings most relevant to the hypothesis. P1.F1, P1.F2 and so on for the findings of P1 |
| Finding description | Plain English, one sentence | Free text |
| Finding quote | The exact text from the paper | Free text, or N/A |
| Finding summary | Structured notation, used to match findings across submissions | For example (APOE3 -> APOE4) -> (less ABCA1), or N/A where the form does not fit |
| Finding relevance | How relevant the finding is to the hypothesis | A number from 0 to 1 |
| Experimental system | The system and the conditions | Free text, as precise as the paper allows |
| Data location | Where the data are | Fig 4C, Table 2, Supplementary Fig 3, or an exact quoted sentence |
| Effect size | The size of the change | A percentage (40%, 300%), a fold change (2.5x), a standardised effect size as the paper gives it (d=3.7, r=0.58), any of these with a leading ~ where you estimated it from a figure (~40%), the word qualitative where the paper shows a direction you cannot put a number on, or N/A |
| Effect size provenance | Where that value came from, or why there is none | One of the codes below |
| P value | The reported significance | A number (0.03), an inequality (<0.001, >0.05), or N/A |
| P value provenance | Where that value came from, or why there is none | One of the codes below |
| Sample size | The number of samples | A whole number (12), a range as the paper gives it (8-12), either with a leading ~ where you counted plotted points instead of reading a stated number (~12), or N/A |
| Sample size provenance | Where that value came from, or why there is none | One of the codes below |
| Directness class | How directly the finding bears on the hypothesis | A number from 1 to 5, on finding rows only |
| Directness strength | How strong that link is | A number from 1 to 5 where 1 is the strongest, or the number with its word (2 strong), on finding rows only |
The directness class
Every hypothesis in this challenge has the same shape. A change in one system variable A causes a change in another system variable B. For the ABCA1 hypothesis, A is the APOE4 variant in place of APOE3, and B is the abundance of ABCA1 protein in the astrocyte outer membrane.
ABCA1 is used on this page and in the example sheet only as a worked example. It is not one of the five challenge hypotheses.
The directness class says how the experiment behind your finding lines up with that shape. It describes what the experiment did, not what the evidence is worth. Use one of:
| Class | The experiment |
|---|---|
| 1 Direct | A itself was perturbed, or varies naturally, and B itself was measured |
| 2 Indirect input | A was reached through a stand-in A’, and B itself was measured |
| 3 Indirect output | A itself was perturbed, and B was inferred from a stand-in B’ |
| 4 Indirect input and output | Both ends go through stand-ins |
| 5 Other | None of the shapes above fit. Say why in Finding description |
Worked example: classes 1 to 4
A worked example, for the hypothesis that APOE4 reduces ABCA1 abundance in the astrocyte outer membrane. Measuring membrane ABCA1 in APOE4 and APOE3 astrocytes is class 1: the genotype is A itself and membrane abundance is B itself. Reducing ABCA1 expression with a microRNA and then measuring membrane abundance is class 2, because the microRNA is a stand-in for the genotype. Comparing APOE4 and APOE3 astrocytes but measuring cholesterol efflux instead of membrane abundance is class 3, because efflux is a stand-in for B. Adding recombinant apoE4 protein to a reporter cell line and measuring efflux is class 4, because both ends are stand-ins.
Three rules settle most of the hard calls.
A perturbation does not have to be introduced by the experimenter. Natural variation counts: comparing APOE4 carriers with APOE3 carriers, or comparing isoforms, is perturbing A. Do not drop a finding to a lower class because the experimenters observed A rather than manipulated it.
The class belongs to the finding, not to the paper. One paper can hold findings of several classes, so the cell is filled on finding rows and left empty on paper rows. The validator rejects a class on a paper row.
The class is relative to the hypothesis you selected. The same experiment would take a different class against a different hypothesis, so do not carry your classes from one submission to another.
The strength of the link. The class says what shape the experiment had. The strength, d in the directness document, says how strong the link is between that experiment and the hypothesis, on a scale of 1 to 5. Note the direction: 1 is the strongest link and 5 the weakest, the same direction as the class, so a low number always means closer to the hypothesis. The template offers the number with its word, 1 very strong through 5 very weak, and a bare number is accepted too.
One question sets it. Could the perturbation have reached the readout by a route that does not pass through the hypothesis? The directness document illustrates this with two experiments on a different ABCA1 hypothesis, that reduced ABCA1 abundance increases Alzheimer’s risk. One reduces ABCA1 expression with a microRNA and measures risk in a mouse model: the microRNA’s other targets are known and none of them look likely to affect risk, so there is no plausible route to risk that goes around ABCA1, and the link is strong. The other uses an LXR/RXR agonist to raise ABCA1 and also measures risk: but the agonist changes APOE expression too, which is an obvious route to risk that never touches ABCA1 abundance, so the same class of experiment gives a much weaker link.
From that question:
| Strength | When |
|---|---|
| 1 very strong | Class 1. A itself and B itself, so there is no stand-in for another route to enter by |
| 2 strong | No plausible route around the hypothesis, and the stand-in is well characterised |
| 3 moderate | A possible route exists and you can argue against it |
| 4 weak | A known, obvious route around the hypothesis |
| 5 very weak | Several such routes, or the stand-in is too far from A or B to say |
Class 4 has a stand-in at both ends, so there are two places an alternative route can enter, and it rarely does better than 3. A class 5 finding is judged on the same question and explained in Finding description.
The validator checks the pair for the one error that is easy to make. A class 1 finding with a strength of 4 or 5 is flagged, because class 1 has no stand-in at either end, so that combination usually means the scale was read upside down.
Publishing your method
A list of findings without a method is a result without a protocol. This challenge is as much about how the literature was read as about what was found, so the method has to be public and it has to be findable.
What to publish. A public repository or archive with a README.md at its root holding six sections:
Who did the work: a person, an agent, a person and an agent, or several agents working together
Model and version, if any was used
How papers were found: sources and queries
How full text was obtained
What a human checked, and on what proportion of the findings
Anything else worth knowing
Copy method_README_template.md from the shared resources page. For work done entirely by hand, six short paragraphs are enough; nobody expects code that does not exist.
What goes in the spreadsheet. The Method sheet carries seven labelled rows. Fill the cell to the right of each label:
| Label | Content |
|---|---|
| Team | Your team name. Use the same one on every submission you make, across all hypotheses |
| Team leader | The person the organizers contact about this submission. A team of one names themselves |
| Team leader email | Where the organizers write about this submission. An institutional or company address where you have one |
| Method URL | The public repository or archive holding your README |
| Commit SHA | The commit your submission was built from, or N/A for an archive with a DOI |
| Who did the work | Exactly one of person, agent, both, collab |
| Model and version | The exact version, one per agent where several were involved. For work done without a model, none or N/A |
The four values for who did the work: person for a human working alone, agent for a single agent, both for a human and an agent, collab for several agents working together. For collab, list every model in Model and version, one per agent, and say in section 1 of your README which agent did which step and whether a person was involved as well.
The two cells have to agree. A submission made entirely by hand is person, and its Model and version is none, or N/A if you prefer the convention used everywhere else in the format. Anything other than person means a model was used, so the model and its version are required. The validator checks the pair in both directions.
Where to publish it. Any public host: GitHub, GitLab, Codeberg, or an archive with a DOI such as Zenodo or OSF. A private repository does not count, and neither does a link that needs an account to open.
Why the commit SHA. A repository can be rewritten, emptied or made private after the deadline, so a bare link is not a record of what produced the submission. The SHA pins it. Run git rev-parse HEAD and put the result in the Commit SHA cell of the Method sheet. An archive with a DOI is already immutable, so it takes N/A instead.
Recording what the paper does not report
A missing number can mean two opposite things. Either the paper never reported it, which is a fact about the literature and one of the things this challenge exists to measure. Or it was reported and the submission did not find it, which is a fact about the submission. Writing N/A for both makes them indistinguishable.
So every statistic carries a provenance code alongside it. Use exactly one of:
| Code | Meaning |
|---|---|
| printed_in_text | The value is printed in the text, a table or a figure legend, and you took it from there |
| visual_only | The paper shows it graphically or as significance stars, with no printed number, and you recorded what it shows without estimating a value |
| estimated_from_figure | There is no printed number and you measured one off the figure, by bar height, axis position or by counting plotted points. The value carries a leading ~ |
| supplement_only | The value is in the supplementary material, and you took it from there |
| not_found_in_source | The paper does not report it anywhere you could find |
| full_text_unavailable | You could not reach the full text, so you cannot say whether it is reported |
| ambiguous_panel | The panel you cite reports several comparisons and it is not clear which one the value belongs to |
Conventions: effect sizes, estimates, P values, significance stars, paywalls, ranges
Report the effect as the paper reports it. A percentage, a fold change and a standardised effect size are all acceptable, and the one to use is the one the authors used. Converting a Cohen’s d into a percentage, or a fold change into a percentage, introduces an assumption that is yours and not theirs. Where the paper gives several, give the one that belongs to the comparison your finding is about.
Estimates are welcome, as long as they say so. Where a figure carries the information but no number is printed, measure it: an effect size from the bars, a sample size by counting the plotted points. Write the value with a leading ~ and use estimated_from_figure. This is better than N/A, because a marked estimate can be used or discarded at will while a blank cannot be recovered. Where you cannot put a defensible number on what the figure shows, use qualitative with visual_only rather than inventing a precision you do not have.
Never estimate a P value. An effect size and a sample size are quantities the figure displays, so measuring them is reading, with an error you have declared by writing the ~. A P value is not displayed. The figure shows the data a test was run on; the test itself happened elsewhere, and its result is either stated or it is not.
So a figure gives you a P value only where the panel or its legend states one. If the panel carries stars, record what the legend says they mean. If it carries nothing, the value is N/A with not_found_in_source. The validator rejects estimated_from_figure on a P value.
Significance stars. Record the inequality the paper’s own legend gives them, with provenance visual_only. The usual convention is * for p<0.05, ** for p<0.01 and *** for p<0.001, but that is only the usual one and it is an example, not the rule: some papers start at ** for p<0.01, some add **** for p<0.0001, and some use the stars for an adjusted P value. Read the legend of the figure you are citing and follow it. Where you can read the key, record the inequality, with provenance visual_only, and do not add a ~: the inequality is what the paper asserts, not something you measured. Where the figure gives no key at all, there is no inequality to record, so the value is N/A with provenance ambiguous_panel. What you must not do is write N/A on a panel whose key you could read.
Paywalls. full_text_unavailable means the full text was genuinely out of reach. Before using it, try the DOI directly, the publisher’s own page, Europe PMC, PMC, and Unpaywall. “My tool could not fetch it” is not the same as “it is not available”, and the two are scored differently.
Ranges and per-group values. Where a paper gives a range (8-12 mice per group), record the range. Do not pick one end of it: a single number that is not in the paper is less faithful than the range that is.
How You’ll Be Evaluated
Unlike most community challenges to date, this one has no ground truth against which to assess the results. Thus, the performance assessment methods break new ground and may evolve as the project progresses. Community challenges differ in the extent to which they predefine assessment criteria. Some, such as those run by Kaggle, predefine a single metric that will be used for ranking. Others, such as CASP, predefine a subset of criteria, but encourage independent assessors to introduce new ones, once the results are in hand. This challenge sets out a set of initial considerations (below), without an explicit ranking formula. These may be modified and added to once the results are analyzed.
Initial assessment will be as follows:
Cross-evaluation by participants to find the subset of strongest performing groups (each participant provides subjective scores for the methods and results submitted by ten other participants, with comments, generating ten assessments per group). Necessary only if there are a large number of submissions.
Comparison of results across groups: which papers were included, which experiments were extracted from those papers. This does not imply that findings reported by one or a few groups will be considered incorrect. Rather, it will help identify ‘easy’ results.
Evaluation of each finding’s relevance using the evidence directness criteria.
Checking of each Finding’s specifics (effect size, P value, experimental system) against the source paper. (Experience with the Hugging Face collab showed that providing these specifics is hard for current methods)
Where necessary, manual inspection.
Submitting
Accepted submissions are given an acceptance number, shown on screen. Keep it. Queries about a submission cannot be handled without its acceptance number.
How to Get Started
Read the hypotheses page and pick one
Download submission_template.xlsx and method_README_template.md from shared resources
Search for papers
Extract findings and fill the spreadsheet as you go, provenance codes and directness included
Validate
Publish your method README, fill in the Method sheet, then upload the spreadsheet and keep your acceptance number
Timeline and Questions
Step 1 opens: 00:00 UTC on November 9th, 2026
First batch of hypotheses released: November 9th, 2026
Step 1 submission deadline: the end of December 20th, 2026, anywhere on Earth
Later phases, evidence assessment, mechanism confidence and experimental design: dates to be announced
Questions go to the Community tab first, then to the organizers. Organizers and the community are here to help.
Thank you for contributing to MecCog. Together we are building a map of APOE4 mechanisms.