Across current random 5-fold runs, best observed performance reaches 62.2% top-1, 80.6% top-3, and 92.8% CSMF accuracy, with interpretation still shaped by strong age-group and cause-class imbalance.
Most important points.
The executive view below is meant for direct sharing. Here, the structured baseline is regularized logistic regression on coded features, the combined model ensembles that baseline with Gemini, and the embedding variants pair narrative embeddings with logistic regression.
| Approach | Top-1 | Why It Matters |
|---|---|---|
| Combined model (structured log-reg + Gemini) | 61.3% | Best current directly matched result and the clearest external headline. |
| Structured model (regularized log-reg) | 56.7% | Strong baseline using coded VA, clinical, and lab-style features. |
| LLM model (Gemini) | 55.6% | Readable and flexible, but not as strong as the combined approach. |
| Age-aware ensemble | 62.2% | Best experimental extension in the current repo. |
| Embedding model (Gemma + log-reg) | 54.8% | Narrative-only model with stronger representations than the smaller embedding run. |
| Model | N | Top-1 | Macro-F1 | Interpretation |
|---|---|---|---|---|
| Combined model (structured log-reg + Gemini) | 9,149 | 61.3% | 0.118 | Best matched result on the direct head-to-head cohort. |
| Structured model (regularized log-reg) | 9,149 | 56.7% | 0.111 | Best structured log-reg baseline in the matched analysis. |
| LLM model (Gemini) | 9,149 | 55.6% | 0.074 | LLM review of serialized patient records. |
| Embedding model (Qwen + log-reg) | 9,149 | 51.0% | 0.032 | Narrative embedding model with logistic regression. |
These rows stay within the same random-split regime but come from slightly different eligible cohorts.
| Variant | N | Top-1 | Top-3 | Top-5 | Macro-F1 | Note |
|---|---|---|---|---|---|---|
| Structured model (full features, regularized log-reg) | 9,542 | 58.9% | 79.4% | 86.4% | 0.088 | Best flat structured log-reg baseline. |
| Structured model (VA + demographics, regularized log-reg) | 9,153 | 56.8% | 77.1% | — | 0.075 | Simpler structured log-reg baseline. |
| Structured model (+ MITS/lab, regularized log-reg) | 9,153 | 56.4% | 78.7% | — | 0.097 | Adds MITS and lab-style signals. |
| Embedding model (Gemma + log-reg) | 9,153 | 54.8% | 75.8% | 84.2% | 0.040 | Best standalone embedding artifact currently in the repo. |
| Embedding model (Qwen + log-reg) | 9,153 | 51.0% | 73.2% | — | 0.032 | Smaller narrative-only model; useful but not the lead result. |
| Combined model (confidence-margin ensemble) | 9,140 | 59.6% | — | — | — | Combined decision rule; +2.9 pp vs the structured log-reg baseline. |
| Age-aware ensemble | 9,542 | 62.2% | — | — | 0.107 | Best overall top-1 in the current random-split artifacts. |
The dataset is broad, but the label distribution is still dominated by a small number of perinatal causes.
The top two causes already make up 54.8% of the cohort.
Stillbirth and neonatal deaths make up most of the current processed dataset.
The core footprint is spread across multiple African sites, with a small Pakistan tail.
Top-1 accuracy on the directly matched comparison cohort.
The combined approach adds value, but the improvements are not uniform. Common neonatal and perinatal patterns are much easier than rarer infectious or mixed categories.
Age groups where the structured log-reg + Gemini ensemble is strongest.
A stepwise view from the leak-safe ablation. Accuracy plateaus around calibration and weight tuning, with the embedding add-on contributing only a marginal final lift.
Counts of cases fixed by the combined ensemble compared with the structured log-reg baseline.
The gain is geographically broad rather than concentrated in a single site.
Darker cells are better within each row.
| Cause | N | Structured model (regularized log-reg) | LLM model (Gemini) | Combined model (structured log-reg + Gemini) | Embedding model (Qwen + log-reg) |
|---|---|---|---|---|---|
| perinatal asphyxia/hypoxia | 3,731 | 88.9% | 84.7% | 88.8% | 90.6% |
| neonatal preterm birth complications | 1,284 | 65.2% | 92.1% | 87.9% | 65.7% |
| congenital birth defects | 721 | 35.6% | 24.5% | 30.7% | 0.3% |
| malnutrition | 478 | 48.1% | 23.4% | 60.0% | 73.4% |
| neonatal sepsis | 437 | 14.9% | 20.8% | 36.8% | 0.0% |
| undetermined | 375 | 4.8% | 5.3% | 5.3% | 0.0% |
| malaria | 331 | 68.0% | 46.2% | 62.5% | 26.6% |
| lower respiratory infections | 294 | 16.7% | 18.4% | 24.8% | 0.0% |
| congenital infection | 203 | 1.0% | 0.0% | 4.9% | 0.0% |
| diarrheal diseases | 165 | 31.5% | 27.3% | 35.2% | 0.0% |
| HIV | 162 | 25.3% | 19.1% | 19.8% | 0.0% |
| other neonatal disorders | 147 | 2.0% | 3.4% | 1.4% | 0.0% |
These are the causes that remain difficult across model families.
| Cause | N | Structured model (regularized log-reg) | LLM model (Gemini) | Combined model (structured log-reg + Gemini) | Embedding model (Qwen + log-reg) |
|---|---|---|---|---|---|
| undetermined | 375 | 4.8% | 5.3% | 5.3% | 0.0% |
| lower respiratory infections | 294 | 16.7% | 18.4% | 24.8% | 0.0% |
| congenital infection | 203 | 1.0% | 0.0% | 4.9% | 0.0% |
| other neonatal disorders | 147 | 2.0% | 3.4% | 1.4% | 0.0% |
| sepsis | 134 | 3.0% | 4.5% | 9.0% | 0.0% |
| neonatal aspiration syndromes | 92 | 3.3% | 3.3% | 3.3% | 0.0% |
| other infections | 69 | 2.9% | 0.0% | 0.0% | 0.0% |
Select a category below to show a quick summary of where the main approaches still struggle.
Common clinical overlap with sepsis and other infections still makes this category hard.
Rare and clinically mixed neonatal infection patterns are still difficult to separate cleanly.
Ambiguous records remain challenging by definition and are unlikely to have a single clean signal source.
Catch-all buckets remain hard for every model family.
Broader sepsis patterns remain difficult outside the highest-volume categories.
These de-identified VA narratives are shown in full because the ambiguity often sits in the complete sequence of symptoms, pregnancy history, delivery course, and care-seeking details rather than in a short excerpt.
Preterm and perinatal asphyxia often share the same narrative signals around gestational age, delivery complications, and a very short neonatal course.
Preterm and perinatal asphyxia often share the same narrative signals around gestational age, delivery complications, and a very short neonatal course.
Preterm and perinatal asphyxia often share the same narrative signals around gestational age, delivery complications, and a very short neonatal course.
Preterm and perinatal asphyxia often share the same narrative signals around gestational age, delivery complications, and a very short neonatal course.
Even detailed pregnancy and delivery narratives can still pull the model toward prematurity when the neonatal decline is brief and unstable.
Even detailed pregnancy and delivery narratives can still pull the model toward prematurity when the neonatal decline is brief and unstable.
Even detailed pregnancy and delivery narratives can still pull the model toward prematurity when the neonatal decline is brief and unstable.
Even detailed pregnancy and delivery narratives can still pull the model toward prematurity when the neonatal decline is brief and unstable.
This longer narrative is a useful counterexample: the model sometimes does stay on the right cause when the symptom pattern is specific enough.
This longer narrative is a useful counterexample: the model sometimes does stay on the right cause when the symptom pattern is specific enough.
This longer narrative is a useful counterexample: the model sometimes does stay on the right cause when the symptom pattern is specific enough.
This longer narrative is a useful counterexample: the model sometimes does stay on the right cause when the symptom pattern is specific enough.
This longer narrative is a useful counterexample: the model sometimes does stay on the right cause when the symptom pattern is specific enough.
These examples are most useful as communication aids. Each transcript is generated by asking the model to rewrite the full structured case into an INTERVIEWER / CAREGIVER exchange that covers symptoms, timing, care-seeking, environmental context, and charted clinical findings without naming the diagnosis.
The full transcript is shown below in a scrollable box rather than excerpted, so the construction and level of detail are visible.
Selected example: site=MZ, age_group=Stillbirth, has_real_narrative=False, words=970.
Selected example: site=BD, age_group=Stillbirth, has_real_narrative=False, words=1,233.
Selected example: site=KE, age_group=Child (12 months to less than 60 Months), has_real_narrative=True, words=1,330.
Selected example: site=MZ, age_group=Death in the first 24 hours, has_real_narrative=False, words=1,230.