• Specified the prompt design and the metadata every item carries: Key Cultural Context, Common Pitfalls, a high/medium/low sensitivity rating and a nation-versus-region scope.
• Built the CAUEF rubric-generation framework and the judge pipeline that consumes it.
• First author of the technical report; ran the client readouts that turned findings into model, task and annotation changes.

ICAB Walkthrough

Where models actually fail

Every rubric constraint is labelled explicit — the prompt spells the requirement out — or implicit, where the model has to infer it from cultural context. Models handle the explicit ones and fail the implicit ones, and the gap is widest exactly where cultural competence lives.

Across all seven languages the worst category is always cultural knowledge or cultural reasoning — never safety, fidelity or helpfulness. Arabic is both the hardest language and the most uneven across providers, with a 23.3-point spread. English has the third highest fail rate, which complicates the assumption that English prompts are a safe default. Region-level prompts fail more than nation-level ones: models lean on the broadest training signal and lose ground when the cultural cue gets specific.

Phase two widens the benchmark on three fronts: more languages and varieties, more modalities, and adversarial prompting. Models generally performed well on the cooperative set, so the open question is whether that holds under red-teaming — when a user is actively trying to push a model into a culturally harmful answer rather than politely asking for one.

• Profiles specify psychological and linguistic targets, which condition generation rather than merely prompting for a persona.
• Outputs measured for linguistic diversity, persona alignment and authorial consistency across prompt conditions.
• Clustering and LIWC embeddings used to test whether the intended profile is recoverable from the text.

Ratings were statistically equivalent across the three strategies, but the failure modes were not. NPS leaned on explicit self-identification — persona role statements in 43.5% of Llama outputs, 43.2% of Mistral, 13.4% of DeepSeek — and read as over-formal. SBS dropped the persona statements entirely and was called “too impersonal” in 31% of cases, mirroring schema keywords literally in 24%. HPS drew the fewest critical comments overall (23% fewer than NPS, 17% fewer than SBS); its own tells were style drift in 25% of outputs and shifting person in 21%.

Across 1- to 4-gram diversity, type/token ratio and compression, condition differences are small and inconsistent while model differences are large: DeepSeek-R1 roughly doubles the 1-gram diversity of Llama 3.3 70B and Mistral Large 2 (0.224 vs 0.126 and 0.111) under identical prompts. Schema steering does shorten outputs sharply — Llama drops from 1,588 characters under NPS to 1,067 under SBS — and lifts type/token ratio, so structure buys concision and vocabulary spread rather than raw diversity.

• Co-led the steerability initiative behind the paper: profile design, generation conditions and evaluation.
• Co-author on the published work.

PILOT: Steering Synthetic Data Generation with Psychological & Linguistic Output Targeting

• Rubrics are binary and atomic: one criterion per check so a failure points at something specific the model did or didn't do.
• Multi-judge by default: two judges score independently and disagreements go to an automated arbiter rather than an average, so a split decision is never silently smoothed over.
• Arbitrated cases are flagged for human review, which is where the interesting failure modes surface — a judge that fails predictably is fixable, one that fails randomly is not.
• Validated at two levels: expert review of a sampled slice of constraints, then a blind human pass scored from scratch with no sight of the judge's verdict.
• Nothing ships on a single agreement number. A pipeline is trusted when it agrees with experts, agrees with blind humans, and repeats itself across passes.

• Instruction sensitivity: how far agreement moves when the rubric is reworded but the criterion is unchanged.
• Judge model choice versus judge instruction quality: which one buys more reliability per unit of effort.
• Whether arbitration should resolve a disagreement or escalate it to a human, and how to tell those cases apart automatically.

• Designed the dual-judge-plus-arbiter architecture and the atomic rubric format it consumes.
• Ran the validation studies and set the thresholds at which a judge pipeline is considered trustworthy enough to report from.
• Carry the practice across projects: each benchmark inherits the previous judge setup and tightens it.

The shape of the work — what a person is handed and what they produce — hints at what the data will be used for. It is evidence, not proof: it narrows the possibilities without settling them, which is why the training pairs it with asking the customer outright.

The people closest to the data see things that typical quantitative analysis can miss: which items are ambiguous, where raters hesitate, which criteria collide in practice. This makes them some of the best-placed people in the pipeline to tell a customer something useful about their model. So I try to teach them to write it up that way: name the pattern, name what it will do downstream, and name the change that fixes it. A percentage says what happened; a percentage with a consequence and a recommendation is insight a customer can act on.

Context beats rules: a rater who knows what the data is for makes better calls on the edge cases no guideline covers. Data sets the ceiling; quality is what separates a good model from an unusable one.

And trust enables insight. I cannot write model insights on data I do not trust, and that trust is what moves the work from data provider to thought partner. Anthropic's own finding that preference raters favoured agreeable answers — and that models trained on those preferences learned to choose agreement over truth — is the cautionary case I teach from: if raters read “helpful” as “agreeable”, that is what we are teaching.