PH.140.850 - Six Ideas in AI Evaluation
A course on how to evaluate AI systems before and after deployment
First Term 2026
Johns Hopkins Bloomberg School of Public Health
Instructor: Yiqun T. Chen
Office: Wolfe Street Building, E3608
Office Hours: Schedule via email
E-mail: yiqunc@jhu.edu
Course Meeting Times: Mondays 3:30–4:50 pm
Location: TBD
Syllabus: Syllabus (PDF)
Course Description
Imagine your manager or collaborator pitches you an AI-based system — whether for ad targeting or for adverse event prediction — and asks how it should be deployed and evaluated. You have a budget, some data, and a deadline. What study do you run, and what evidence would make you say go or no-go?
This course is about starting to answer that question. We will not cover the details of any AI method, but rather their high-level properties — stochasticity, bias, a fast product life cycle — and how those properties bear on the design of an evaluation. The six topics are prediction evaluation, benchmarks, model-generated labels, study design, subgroup performance, and distribution shift. They are ordered so that the course moves from a static picture to a dynamic one: the first five weeks treat an evaluation as a measurement taken once, and the last week asks what happens when the population moves and the vendor ships an update. Each topic is taught on small semi-synthetic datasets where the truth is known, so that failure modes can be seen directly rather than described.
This is Part I of a two-part sequence and focuses on evaluation: what to measure, how to measure it, and what the resulting numbers do and do not license you to claim. Part II will build on this foundation and cover intervention and decision-making, dynamic and interactive evaluation of agentic systems, and other advanced topics.
Intended Audience: Master’s and PhD students in Biostatistics or other departments with quantitative training, or by instructor permission.
Prerequisites: Familiarity with regression and basic inference. Students should have used an LLM to solve a technical problem, broadly defined. Some programming experience in Python or R, or a willingness to learn — including by vibe coding. No prior exposure to deep learning is assumed.
Resources
Course Materials: Slides, notebooks, datasets, and recommended readings will be posted on this website.
Schedule (Tentative)
| Week | Date | Topic | Materials |
|---|---|---|---|
| 1 | Aug 31 | Evaluating predictions. Accuracy and calibration; classification and calibration metrics; proper scoring rules; and the behavior of these metrics under low prevalence and class imbalance. | Notes (PDF) |
| - | Sep 7 | Holiday (Labor Day) - No class | - |
| 2 | Sep 14 | Benchmarks. What is a benchmark, and what is it measuring? Construct validity, contamination, leaderboards, and Goodhart’s law. Steps for building an evaluation set for a system nobody has benchmarked. | TBA |
| 3 | Sep 21 | Model-generated labels. Human labels are expensive, so AI models are increasingly used to augment them. Measurement error, prediction-powered inference, and the reliability of LLM annotators. | TBA |
| 4 | Sep 28 | Designing the validation study. Earlier weeks examine data someone else has already collected; this week we plan our own. Power analysis, active learning, and pre-registration. | TBA |
| 5 | Oct 5 | Subgroups, fairness, and privacy. Does the deployed system work equally well for everyone? Fairness criteria, subgroup analysis, pooling when groups are small, and how privacy comes into play. | TBA |
| 6 | Oct 12 | Distribution shift. The first five weeks treat the evaluation as a measurement taken once. Populations move, sites differ, and vendors ship updates. Diagnosing degradation across time and sites, and turning a one-time validation into a monitoring plan. | TBA |
Evaluation and Grading
Pass/Fail. Three short assignments, each covering two topics.
| Assignment | Covers | Released | Due |
|---|---|---|---|
| 1 | Validity, prediction, and benchmarks (Topics 1–2) | Sep 14 | Sep 28 |
| 2 | Model-generated labels and study design (Topics 3–4) | Sep 28 | Oct 12 |
| 3 | Subgroups, fairness, and distribution shift (Topics 5–6) | Oct 12 | Oct 19 |
Assignments run on a single case study that is threaded through all six topics. Each mixes calculation with a short piece of writing addressed to a non-statistician, because both are part of the job.
Learning Objectives
Upon completion of this course, students will be able to:
- Evaluate a prediction system.
- Judge what a benchmark score does and does not imply about a deployment.
- Draw valid conclusions when outcomes are labeled by a model (e.g., an LLM) rather than by an expert gold standard.
- Design a validation study with an explicit sampling plan and budget.
- Assess subgroup performance and the trade-offs among fairness criteria.
- Diagnose whether and why performance degrades across sites and populations, and specify the monitoring that follows deployment.