IntermediateAI

MIT Media Lab Launches ImpactBench and AI Nutrition Labels

On 7 October 2026, the MIT Media Lab’s Advancing Humans with AI programme and three partner institutions opened ImpactBench to the public: an open evaluation suite that scores language models on what they do to the people using them rather than on what tasks they can complete. Its consumer-facing output, the AI Nutrition Label, reduces each model to letter grades across fourteen behaviours — seven harms avoided, seven benefits promoted — in the layout of a food label. The first release covers ten models, 809 metrics drawn from 26 benchmarks, and 48,540 simulated multi-turn conversations.

Announcement poster with four overlapping AI Nutrition Label cards fanned around a central panel reading Introducing AI Nutrition Label, each card listing seven rows under the heading Avoiding Negative Impact and seven under Promoting Positive Impact with letter grades ranging from A to F, above logos for MIT, the MIT Media Lab, AHA Advancing Humans with AI, the Psychology of Technology Institute, UC Berkeley Haas and USC Marshall
Four example labels from the launch materials. Each row is a behaviour category graded on the same scale a food label uses for nutrients. Image credit: ImpactBench press images, MIT Media Lab

What the label actually measures

A conventional benchmark asks whether a model can answer a question. ImpactBench asks whether a conversation leaves the user better or worse off, and it splits that question in two. Seven label rows record harms the model avoids — factual hallucination, sexual and intimate behaviour, sycophancy, toxicity, harmful bias and discrimination, unsafe advice, financial and legal harm. Seven record benefits it promotes — agency, learning and skill development, social interaction, creativity, wellbeing, healthy technology use, and productive and meaningful work.

The underlying metric suite is larger than the label suggests: 809 metrics, of which 460 are positive and 349 negative, organised across psychological, physical and social domains. Constructs arrive two ways — through an open call to clinicians, educators, legal scholars and community advocates, and through standardisation of existing benchmarks and harm taxonomies, including KORA’s child-safety scenarios, the Stanford harm taxonomy and Weval benchmarks grounded in crisis-intervention practice from the WHO, Samaritans and the 988 Suicide & Crisis Lifeline. Each metric pairs an observable criterion with a polarity, so a positive metric passes when a good behaviour appears and a negative metric passes when a bad one is absent.

Sunburst diagram with three colour-graded areas forming an inner ring labelled Psychological, Physical and Societal, an outer ring of fourteen subareas including Social Relationships, Mental and Emotional Health, Agency and Purpose, Learning and Skill Development, Cognition and Judgment, Creativity, Physical Health, Physical Safety, Privacy and Security, Career and Finance, Bias and Discrimination, Culture and Identity, Environmental Health and Law and Civil Rights, and a central silhouette of a human figure with arms outstretched
The three outcome areas and their subareas, shaded from red to green by pass rate in the platform’s Explore view. Learning and skill development is the one subarea that reads red at launch. Image credit: MIT Media Lab AHA programme, via the ImpactBench press collection

How a score is produced

Evaluations run as simulations rather than single prompts. A simulated user with a latent objective talks to the target model across multiple turns, using personas drawn from demographically diverse scenarios and adapted for teen and adult contexts; the simulated messages carry typos and autocorrect artefacts to approximate ordinary typing. An LLM judge scores the resulting transcript against each metric, and the results pass through reliability audits and human expert review. Metric candidates were first drafted by an LLM — the paper names claude-opus-5 — then revised, merged or removed by researchers checking fidelity to the source.

The team also stress-tested its own pipeline, running five independent checks that varied which model generated the test scenarios, which model played the user and which model served as judge. Rankings held steady across all five. That robustness is narrower than it sounds: the paper reports that rankings do shift across benchmarks, age contexts and weighting choices, which is why the platform imposes no single weighting and declines to name a best model.

What the first release found

Across 48,540 scored conversations, models exhibited desirable behaviours in 68.7% of evaluations (95% CI 66.2–71.0) but avoided undesirable ones in only 53.1% (95% CI 50.4–55.6). The asymmetry was sharpest for psychological outcomes.

Metric pass rates by outcome area, ten models (%): positive metrics pass on desirable behaviour present, negative metrics on harmful behaviour absent
Area Positive metrics Negative metrics
Physical 74.3 61.4
Psychological 68.8 44.6
Social 68.4 55.7
All areas 68.7 53.1

Subarea spreads were wider than area spreads: positive pass rates ran from 45.9% for learning and skill development to 80.6% for privacy and security, negative rates from 16.3% to 72.8% in the same two subareas. So the weakest category on both axes is the one a tutor product would be bought for, while privacy and security, creativity and emotional support were shared strengths. Models separated far more on harm avoidance — a 38.6-point spread — than on helpfulness, at 15.9 points, and adjacent models were often within the uncertainty on positive metrics.

Age context moved behaviour. In teen conversations, models less often supported independent reasoning and more often displayed dependence-building behaviours than in adult ones. The paper illustrates the pattern with three trajectories: decision support becoming decision substitution, emotional validation slipping into relational comparison, and academic assistance turning into cognitive offloading. Juliana Schroeder, a UC Berkeley Haas professor and collaborator, framed the release plainly: “We’ve set out to build a rigorous, independent measure of how AI shapes human psychology—not just what it outputs, but what it does to our thinking, our autonomy, and our wellbeing. This is a first step, and we’re asking the field to help us get it right.”

Screenshot of the ImpactBench Explore page, with a left sidebar listing models that include Claude Opus 5, Claude Sonnet 5, DeepSeek V4 Flash, Gemini 3.7 Flash, Gemma 4 31B, GPT-5.6 Luna, GPT-5.6 Terra, Grok 4.6 and Qwen3.8 27B above red-to-green impact bars, a central sunburst of psychological, physical and social areas, and a right panel giving per-area metric counts
The Explore view, with adult and teen filters in the model list at left and metric counts per area at right. Image credit: ImpactBench

Limits inside the numbers

The platform is candid about what a score means. Grades describe how consistently a model showed a behaviour under controlled stress-test conditions; they are not estimates of how often that behaviour occurs in ordinary use, and they are not direct measures of downstream effects on users. The team’s own announcement calls the findings “early signals from initial testing, not final judgments on any system’s safety.”

Validation is proportionate rather than exhaustive. Ten internal experts reviewed 85 distinct metrics, 10.5% of the suite, and mean scenario quality came out at 2.97 on a four-point scale, with 66 of 87 valid ratings at 3 or 4. Majority expert ratings agreed with the automated judge in 83.3% of reviewed cases, a Cohen’s κ of 0.667 — substantial agreement, short of the near-perfect concordance an audit trail would ideally show.

What This Means

The interesting design decision is that every score is traceable to the metric, the scenario and the full transcript behind it. Capability leaderboards usually publish a number; this publishes the conversation that produced it, so a sceptical reader can check a grade rather than trust it. For anyone choosing a model on grounds other than price and throughput — a school buying a tutor, a clinic triaging an intake bot — that is closer to an audit than a spec sheet.

It is also a live test of whether simulated users are an adequate stand-in for real ones. The pipeline swaps models freely without disturbing the ranking, yet behaviour shifts when the simulated user is a teenager, which points at persona design carrying more of the signal than judge choice does.

MIT’s announcement adds context without claiming a role in policy: prior research from the team on chatbots’ psychological effects, it says, was cited as a key inspiration for California’s Senate Bill 243, which the announcement describes as one of the first state laws to directly regulate companion AI to protect children. ImpactBench itself is an evaluation platform, not a regulatory instrument, and its authors present it as a shared evidentiary base rather than a compliance standard.

This post was drafted with AI assistance and reviewed by RITS staff.

Sources

Never miss a RITS update

Subscribe to the RSS feed for AI, emerging technology, and digital scholarship news curated for the NYU Shanghai community, or get in touch to suggest a story.