Beyond journals: open evaluation

What The Unjournal does, what’s new in 2026, and how evaluation can feed replication

David Reinstein · Co-director, The Unjournal · unjournal.org

Open Science Futures in Social & Behavioral Sciences · Monk Prayogshala × Institute for Replication × The Unjournal · 13 November 2026

In one slide

The Unjournal is a nonprofit that pays experts to publicly evaluate and rate research that is already public: working papers, preprints, reports, and other research objects, including dynamic documents and interactive models.

No accept/reject. No fees. Reports, ratings and author responses are open and citable.

Focus: economics, policy and quantitative social science that could matter for global wellbeing.

Two of us are from Monk Prayogshala: Anirudh Tagat (co-director) and Hansika Kapoor (management team).

1 · Why not “just use journal peer review”?

The scarce part is the evaluation

Journals do real things: curation, a trusted signal.

But we already share our own work: preprints, working papers.

The valuable part, expert judgement, ends up as a private yes/no.

Sent sequentially to 3–4 journals: years of delay, worst for timely research.

The publish-or-perish slot machine

★ THE PUBLISH-OR-PERISH SLOT MACHINE ★
insert: 1 finished paper · ~6 months / spin
✗↻✓
rejectR&Raccept
PAYOUT: one line on your CV
Each spin takes ~6 months; placement often takes 2–6 years.

“Published, so stop bothering me about it”

Journals take one format: ~30 static pages.

Publication says “done”, so we move on to the next paper.

Little room to improve, correct, or build in place.

Research as a living document

Separate evaluation from the journal, and research can live in better formats:

Any format: notebooks, dashboards, interactive and replicable documents.

Improve in place, then ask for further evaluation.

Open evaluation feeds open science and replication: reviewers can point at code and data, not just prose.

An interactive specification-curve (“multiverse”) document: easy to build now with AI-assisted coding, and far easier to scrutinise than static pages.

Separating evaluation from publishing

2 · What The Unjournal does

unjournal.org  ·  unjournal.pubpub.org

We are not a journal

We commission open evaluation of public research.

Paid experts, usually two per paper (sometimes more or fewer).

Reports, ratings and an author response: public, with DOIs.

Not tied to any journal: authors can still publish anywhere.

Grant funded

What an evaluation gives you

We don’t accept/reject or assign a tier, so we benchmark instead. Two halves, equally important:

The substance

  • Detailed referee-style reports
  • The authors’ response
  • An editorial summary

The structured ratings

  • Percentile ranking on several criteria, including open science
  • Journal-tier equivalent, “should” and “will”
  • Every rating with a 90% credible interval
  • Claim identification and assessment

All public and comparable · see the evaluator form →  ·  ↓ the instrument

Inside the evaluation form

Each rating comes with a 90% credible interval, not just a number.
Percentile rating  ·  Methods: justification, reasonableness, validity, robustness
0255075100
Midpoint 72  ·  90% CI 66 – 78  (illustrative)
Claim identification & assessment: evaluators identify the paper’s key claims and rate, for each, the strength of evidence, and they can suggest specific robustness checks.
Open the evaluator form →

All the rating criteria, and what we’ve seen so far

Criterion Scale Median rating Papers
Overall assessment 0–100 77 61
Claims, strength and characterization of evidence 0–100 80 30
Methods: justification, reasonableness, validity, robustness 0–100 75 60
Advancing knowledge and practice 0–100 80 60
Logic and communication 0–100 79 61
Open, collaborative, replicable research 0–100 80 61
Real-world relevance (earlier form) 0–100 84 56
Relevance to global priorities 0–100 85 61
Journal tier it should reach (merit) 0–5 4.0 54
Journal tier it will reach (prediction) 0–5 4.0 52
Median of evaluators’ midpoints across our public data; “Papers” = papers with that rating. The two journal-tier ratings separate merit (“should”) from predicted placement (“will”).  Full dashboard →

Journal tier: should vs will

Evaluators say which journal tier (0–5) the paper should reach on merit, and which it will reach. A bridge to the signal committees already use.

How it works: one paper, start to finish

Example: Water Treatment and Child Mortality (meta-analysis) · hover the underlined text for what happened with this paper

  1. Suggested and prioritised by the team: high stakes for funders
  2. An evaluation manager is assigned: like a handling editor
  3. The manager recruits paid evaluators (here two, one team with systematic-review expertise) and briefs them
  4. We tell the authors and invite them to engage
  5. Evaluators submit reports + ratings: flagged real issues
  6. Authors respond publicly, and revised their framing
  7. Package published with DOIs (Sept 2024): GiveWell: “very influential”
Target: ~2–3 months from selection to publication.  Read the package →

What the evaluators said

Water Treatment and Child Mortality: from the published evaluations

“Major concerns about conduct and reporting, which did not follow systematic review standards of transparency.”Evaluator 1 (signed; a team with systematic-review expertise). They found data-extraction differences worth around a third of the pooled effect.
“I do not see any major technical issues… the ‘cost’ part of the paper could be deepened.”Evaluator 2 (anonymous; focused on the cost-effectiveness analysis)

The published package

Overall: 50 vs 85.

Journal tier: 2.5 vs 3.7.

Both with 90% intervals, plus authors’ response.

unjournal.pubpub.org/pub/evalsumwatertreatment · DOI 10.21428/d28e8e57.c4f3e6dd

How is this different from other open peer review?

Other open review The Unjournal
Who reviews Volunteers, commenters, or journal referees whose reports are published Commissioned, paid experts, matched to each paper
Which papers Whatever is submitted or draws attention Prioritised for decision-relevance
What you get Prose reports Reports + structured ratings with uncertainty + claim assessment
Tied to a journal? Usually part of one journal’s process No: any public paper; authors can still publish anywhere
Who it’s for Mainly authors and editors Authors, and research users: funders, policymakers

Our workflow: from paper to evaluation package

Research identified or submitted
→
Prioritise for impact potential
→
Manager recruits paid experts (usually two)
→
Reports + structured ratings
→
Authors respond; manager synthesises
→
Evaluation
package
published
AI helps curate and suggest research
AI flags potential issues in the paper, shared with evaluators
AI checks evaluators’ reports: did they miss something?
Most work we assess is deprioritised at step 2.

How the triage runs

Rapid path (AI economics): public prioritisation page → two strong yes votes → a week for comments → go

Where we are now

58+ evaluation packages published

100+ expert evaluations

180+ evaluators in the pool

23 packages in global health & development

1,000+ structured ratings

40+ field specialists

0 fees for authors or readers

Including evaluations of cash transfers for child development in India and the India–UAE guest-worker experiment.  ·  Founded 2022 · unjournal.pubpub.org · ratings dashboard

Every rating comes with a credible interval

0 25 50 75 100 Overall rating (0–100 percentile), mean across evaluators; bar = mean 90% interval. Hover a row for the paper. Replicability & Generalisability: A Guide to CEA discounts Overall: 32 (90% interval 5–55); 2 evaluator(s) Zero-Sum Thinking, the Evolution of Effort-Suppressing Beliefs, and Economic Development Overall: 40 (90% interval 30–50); 1 evaluator(s) Pharmaceutical Pricing and R&D as a Global Public Good Overall: 40 (90% interval 22–58); 2 evaluator(s) Global potential for natural regeneration in deforested tropical regions Overall: 50 (90% interval 35–70); 2 evaluator(s) Accelerating Vaccine Innovation for Emerging Infectious Diseases via Parallel Discovery Overall: 50 (90% interval 35–65); 2 evaluator(s) Meaningfully reducing consumption of meat and animal products is an unsolved problem: A meta-analysis Overall: 57 (90% interval 37–71); 2 evaluator(s) Social Safety Nets,Women’s Economic Achievements and Agency Overall: 60 (90% interval 50–70); 1 evaluator(s) Irrigation Strengthens Climate Resilience: Long-term Evidence from Mali using Satellites and Surveys Overall: 60 (90% interval 58–66); 2 evaluator(s) Long term cost-effectiveness of resilient foods for global catastrophes compared to artificial general intelligence safety Overall: 62 (90% interval 48–72); 3 evaluator(s) Biodiversity risk Overall: 63 (90% interval 50–76); 2 evaluator(s) Observational price variation in scanner data cannot reproduce experimental price elasticities Overall: 65 (90% interval 48–80); 2 evaluator(s) The Governance Of Non-Profits And Their Social Impact: Evidence From A Randomized Program In Healthcare In DRC Overall: 65 (90% interval 55–74); 2 evaluator(s) Deadweight Losses or Gains from In-kind Transfers? Experimental Evidence from India Overall: 65 (90% interval 55–75); 1 evaluator(s) Water Treatment and Child Mortality: A Meta-analysis and Cost-effectiveness Analysis Overall: 68 (90% interval 58–78); 2 evaluator(s) The Effect of Public Science on Corporate R&D Overall: 70 (90% interval 66–78); 2 evaluator(s) Stagnation and Scientific Incentives Overall: 70 (90% interval 55–85); 1 evaluator(s) Does Conservation Work in General Equilibrium? Overall: 70 (90% interval 60–80); 1 evaluator(s) Effects of Emigration on Rural Labor Markets Overall: 70 (90% interval 50–78); 1 evaluator(s) A review of GiveWell’s discount rate Overall: 71 (90% interval 52–84); 2 evaluator(s) Population ethical intuitions. Overall: 71 (90% interval 58–87); 2 evaluator(s) Choose Your Moments: [NIH] Peer Review and Scientific Risk Taking Overall: 72 (90% interval 55–85); 2 evaluator(s) Towards best practices in AGI safety and governance Overall: 72 (90% interval 62–82); 2 evaluator(s) How Much Would Reducing Lead Exposure Improve Children’s Learning in the Developing World? Overall: 72 (90% interval 62–78); 2 evaluator(s) When Celebrities Speak: A Nationwide Twitter Experiment Promoting Vaccination In Indonesia Overall: 74 (90% interval 74–74); 2 evaluator(s) Technical Report: The Value of Existential Risk Mitigation Overall: 75 (90% interval 70–80); 1 evaluator(s) Urban Forests: Environmental Health Values and Risks Overall: 75 (90% interval 66–92); 2 evaluator(s) Cognitive Behavioral Therapy among Ghana’s Rural Poor Is Effective Regardless of Baseline Mental Distress Overall: 75 (90% interval 72–80); 2 evaluator(s) Legalizing Entrepreneurship Overall: 75 (90% interval 68–84); 2 evaluator(s) The Benefits and Costs of Guest Worker Programs: Experimental Evidence From the India-UAE Migration Corridor Overall: 75 (90% interval 65–88); 2 evaluator(s) Banning wildlife trade can boost demand for unregulated threatened species Overall: 75 (90% interval 75–75); 2 evaluator(s) The Returns to Science In the Presence of Technological Risks Overall: 76 (90% interval 44–88); 2 evaluator(s) Social Safety Nets, Women’s Economic Achievements and Agency Overall: 77 (90% interval 71–80); 1 evaluator(s) Ends versus Means: Kantians,Utilitarians,and Moral Decisions Overall: 78 (90% interval 60–90); 2 evaluator(s) Forecasts estimate limited cultured meat production through 2050 Overall: 78 (90% interval 60–86); 2 evaluator(s) Maternal cash transfers for gender equity and child development: Experimental evidence from India Overall: 78 (90% interval 72–82); 2 evaluator(s) Forecasting Existential Risks: Evidence From a Long-Run Forecasting Tournament Overall: 78 (90% interval 70–85); 2 evaluator(s) The Environmental Effects of Economic Production: Evidence from Ecological Observations Overall: 79 (90% interval 79–79); 2 evaluator(s) Advance Market Commitments: Insights from Theory and Experience Overall: 80 (90% interval 70–88); 3 evaluator(s) Asymmetry in Civic Information: An Experiment on Tax Participation among Informal Firms in Togo Overall: 80 (90% interval 75–85); 2 evaluator(s) Adaptability and the Pivot Penalty in Science Overall: 80 (90% interval 70–90); 1 evaluator(s) The animal welfare cost of meat: evidence from a survey of hypothetical scenarios among Belgian consumers Overall: 80 (90% interval 70–90); 1 evaluator(s) Economic vs. Epidemiological Approaches to Measuring the Human Capital Impacts of Infectious Disease Elimination Overall: 80 (90% interval 65–90); 1 evaluator(s) Existential risk and growth Overall: 82 (90% interval 70–84); 2 evaluator(s) How Effective Is (More) Money? Randomizing Unconditional Cash Transfer Amounts in the US Overall: 82 (90% interval 72–92); 2 evaluator(s) The Comparative Impact of Cash Transfers and a Psychotherapy Program on Psychological and Economic Well-being Overall: 82 (90% interval 78–88); 2 evaluator(s) The Long-Run Effects of Psychotherapy on Depression, Beliefs, and Economic Outcomes Overall: 84 (90% interval 76–92); 2 evaluator(s) Does the Squeaky Wheel Get More Grease? The Direct and Indirect Effects of Citizen Participation on Environmental Governance in China Overall: 85 (90% interval 78–90); 2 evaluator(s) When do “Nudges” Increase Welfare? Overall: 86 (90% interval 73–96); 2 evaluator(s) The wellbeing cost-effectiveness of StrongMinds and Friendship Bench: Combining a systematic review and meta-analysis with charity-related data Overall: 86 (90% interval 80–92); 2 evaluator(s) Artificial Intelligence and Economic Growth Overall: 86 (90% interval 75–95); 2 evaluator(s) Misperceptions and Demand for Democracy under Authoritarianism Overall: 86 (90% interval 75–95); 2 evaluator(s) Building Resilient Education Systems: Evidence from Large-Scale Randomized Trials in Five Countries Overall: 86 (90% interval 79–94); 2 evaluator(s) The Macroeconomic Impact of Climate Change: Global vs. Local Temperature Overall: 88 (90% interval 80–95); 2 evaluator(s) Money (Not) to Burn: Payments for Ecosystem Services to Reduce Crop Residue Burning Overall: 88 (90% interval 78–92); 2 evaluator(s) Intergenerational Child Mortality Impacts of Deworming: Experimental Evidence from Two Decades of the Kenya Life Panel Survey Overall: 88 (90% interval 78–98); 2 evaluator(s) Willful Ignorance and Moral Behavior Overall: 89 (90% interval 79–94); 2 evaluator(s) Universal Basic Income: Short-Term Results from a Long-Term Experiment in Kenya Overall: 89 (90% interval 74–98); 2 evaluator(s) Selecting the Most Effective Nudge: Evidence from a Large-Scale Experiment on Immunization Overall: 90 (90% interval 78–94); 2 evaluator(s) A Welfare Analysis of Policies Impacting Climate Change Overall: 90 (90% interval 82–94); 2 evaluator(s) Does online fundraising increase charitable giving? A nationwide field experiment on Facebook Overall: 91 (90% interval 85–95); 2 evaluator(s) Adjusting for Scale-Use Heterogeneity in Self-Reported Well-Being Overall: 95 (90% interval 85–100); 2 evaluator(s)
One row per paper with ratings in our public data (61 papers, 113 overall ratings). Hover a row to see the paper.  ratings dashboard →

Comparing two papers across criteria

One number hides too much: two evaluated papers compared across the rating criteria.  Explore in the dashboard →

An example: an award-winning evaluation

2024–25 Evaluator Prize · 1st Evaluation of “Water Treatment & Child Mortality: a meta-analysis”

“Very influential.”GiveWell’s water team; the evaluation informs chlorination grantmaking.
“Thorough and thoughtful… extensive write-up and precise recommendations.”The paper’s authors, who revised the framing in response.

Read it →

3 · What’s new in 2026

Faster, more focused on decisions, and AI-assisted

Three shifts this year

Focused streams

AI economics & governance: ~15 evaluations over 12 months.

Public signal within about a month: this work goes stale fast.

From papers to decisions

Pivotal Questions: start from a decision someone faces; elicit experts’ quantified beliefs.

Recent examples.

AI-assisted, human-led

AI helps find, triage and check.

Humans decide what matters.

We test AI reviewers against human ones.

Pivotal Questions: what expert beliefs look like

n = 10 expert forecasts from our cultivated-meat workshop: medians with 80% credible intervals.  Pivotal Questions →

Metrics and LLM tools: what we’re learning

Exploratory · preliminary

Ratings: LLM and humans only modestly agree.

Critiques: a one-shot model raised 35–61% of an evaluator’s issues; a second evaluator, 22–36%.

Already useful as a first pass, and it may take over some of the work.

We also audit ourselves.

Methods and results (in progress): llm-uj-research-eval.netlify.app

AI makes evaluation more important

A flood of plausible AI-generated papers: some correct and useful, many not.

So more need for efficient, transparent evaluation connected to people who use research.

AI can help with code and data checks, where authors share them.

Keep a human in the loop for the judgement calls.

Not “does it fit a top-journal template?” but “is it true, and does it matter?”

4 · Evaluation + replication

“Evaluation games”

Like a journal club, but producing something public.

4–5 researchers, one paper, read in advance.

Meet and fill in our structured form (or an adapted version) together.

Focus on key claims and specific robustness checks.

Output: a public evaluation + a notebook of checks.

From evaluation to replication

1 · Evaluate
Which claims matter most? How strong is the evidence? Do the assumptions, theory and policy or field context hold up? Which robustness checks, specified in detail, would be most informative?
→
2 · Replicate
Replicators reproduce the results (as in I4R’s Replication Games), run the suggested checks and their own, and note which they used or rejected, and why. Some people can be on both teams.
→
3 · Public record
Evaluation, replication and author response, linked and citable, with both teams credited; perhaps a live multiverse or dashboard page.

Evaluators bring judgement about what matters; replicators bring code and data.

The notebook travels with the paper; software barriers are shrinking.

We’ve been talking with I4R about this since 2022. Lenka says more next.

Software barriers: Stata and code conversion

Many replication packages use Stata, which needs a paid licence.

AI tools can convert most Stata code to R or Python.

Some Stata-specific procedures are hard to rebuild elsewhere, even with AI.

Always check converted output against the original.

5 · Getting involved

For researchers and students in South Asia

Evaluate, with modest pay: postdocs, faculty and advanced PhD students; named or anonymous.

Evaluation or replication games: thanks for telling us your interest when you registered.

Suggest research that deserves careful public scrutiny, including work from and about South Asia.

Become a field specialist: help choose what we evaluate in your area.

Use our outputs: evaluations for teaching and reading groups; the ratings data for meta-science.

Ask for evaluation of your own public work: credible feedback that doesn’t depend on where you sit.

unjournal.org · join the evaluator pool · contact@unjournal.org

Students & early-career researchers

See what researchers, funders and practitioners actually care about: it sharpens your own work.

Do real peer review and get feedback on your evaluation, from us and often from the authors.

Join a network of funders and impact-focused researchers.

Reading groups can shadow-evaluate a published package and compare with the expert evaluations.

Questions for the panel, and for you

  • What would make a public evaluation count for hiring and promotion here?
  • What gets in the way of open practice: data access, software, time, incentives?
  • How should evaluation and replication feed each other?
  • Where should AI tools help with review, where shouldn’t they, and why?

Thank you

Open evaluation, now: faster feedback and a credible public signal.

Together with replication: checks on the claims that matter.

Your part: evaluate, suggest research, join a game, use the data.

David Reinstein · contact@unjournal.org · unjournal.org · unjournal.pubpub.org

1 · Why 2 · What we do 3 · New in 2026 4 · + Replication 5 · Get involved