Case study 01 · LLM behavior evaluation
Human-Grounded LLM Evaluation
I examine how real households decide whether to buy flood insurance, then test whether AI-generated survey answers preserve those decision relationships.
- real household survey evidence
- overall and household-constraint group comparisons
- repeated responses, not additional participants
Why this problem matters
If AI-generated answers are used to inform flood-adaptation support, sounding realistic is not enough. We need to know whether they reflect household differences and the relationships behind willingness to buy insurance.
How I approached it
I analyzed household surveys from the Passaic River Basin and compared them with responses from two LLMs using matched household profiles.
What it contributes
Human-grounded evaluation reveals differences in household variation and decision relationships that comparisons of average answers alone can miss. Plausible generated responses are not a substitute for checking against human evidence.
The same problem appears in simulated-user research, customer agents, health-risk assessment, and policy modeling: an output can sound human without behaving consistently with measured evidence.
System artifact
Inspect the research logic
The public diagram is a study-design schematic. Numeric comparison results remain unpublished and are intentionally omitted.
Evidence enters. Decisions stay inspectable.
| Direction | Reference | Model choices |
|---|---|---|
| Low efficacy | 0 / 4 | 4 / 4 |
| High efficacy | 4 / 4 | 0 / 4 |
The direction reverses: higher efficacy predicts action in the reference, but inaction in model choices.
Record A1Reference: WaitModel runs: Act / Act / Act
Action counts in authored fixtures; not study estimates.
Interactive controls require JavaScript. The complete flow and sources remain below.
Evaluation
- Human evidence
Measured constructs and reported choices establish the empirical reference.
Published evidence - Context
A label-blind synthetic profile carries only approved attributes into the prompt.
Illustrative example - LLM proposal
Repeated model runs produce a choice and a concise rationale for comparison.
Illustrative example - Validation / repair
Checks compare direction, subgroup behavior, and run-to-run stability without exposing respondent records.
Public artifact - Consequence
The result becomes an evaluation finding, not a claim that the model represents a person.
Illustrative example
Governance
- Human evidence
Behavioral and physical constraints define what a plausible action must respect.
Published evidence - Context
A synthetic agent state limits the actions the model may propose.
Illustrative example - LLM proposal
The LLM returns a structured proposal rather than mutating the simulation directly.
Illustrative example - Validation / repair
Validators reject invalid fields, request targeted repair, and retain an audit trace.
Public artifact - Consequence
Only an accepted proposal updates the coupled model state.
Illustrative example
Simulation
- Human evidence
Survey evidence and public flood-risk sources ground household behavior.
Published evidence - Context
Household tenure, risk perception, resources, and exposure form the decision context.
Published evidence - Agent decision
Agents choose adaptation, insurance, or no action under modeled constraints.
Public artifact - Validation / repair
Model checks keep behavioral, financial, and physical state transitions consistent.
Public artifact - Consequence
Flood losses and adaptation alter the household and environmental state for the next cycle.
Published evidence
Research context and method
Study-design schematic · no unpublished coefficients
What this lens emphasizesCompare the full pathway structure against the survey-grounded reference.
Method in detail
- Build household profiles from demographic, socioeconomic, housing, and flood-related information.
- Have both LLMs answer the same survey questions, with repeated runs under unchanged settings.
- Use Confirmatory Factor Analysis to check measurement quality separately for human and generated responses.
- Use Structural Equation Modeling to compare decision relationships overall and across household-constraint groups.
Validation and limitations
Validation
- Separate measurement-quality checks for human and generated responses
- Overall and household-constraint group comparisons
- Repeated runs with unchanged settings
- Distinguish averages, household variation, and decision relationships
Limitations
- One regional, cross-sectional survey cannot establish generalizability across places or time.
- Decision pathways describe associations, not causal effects.
- Willingness to buy insurance is not observed purchasing behavior.
- Generated responses are synthetic, not additional human participants.
Evidence slice
From question to consequence
- 01Problem
- Can generated flood-insurance survey answers preserve the household differences and decision relationships found in real human responses?
- 02My role
- I compared measured household survey responses with two LLMs using matched profiles, measurement checks, decision-pathway analysis, and repeated runs.
- 03Input
- Real household survey evidence and generated answers to the same questionnaire using household context.
- 04Decision
- Compare willingness to buy insurance and the relationships between measured decision variables.
- 05Validation
- Check measurement quality separately, compare household-constraint groups, and test repeated responses under unchanged settings.
- 06Consequence
- Reveal differences in household variation and decision relationships that average answers alone can hide.
SourcesRelated research profile
Contact
Build evaluations that connect model behavior to real evidence
Interested in evaluating AI decisions or building evidence-linked research systems? I welcome research and open-source collaboration, and Summer 2027 internship conversations.
More about my research path