TurnBench: A Multi-Domain Benchmark for Turn-Taking Dynamics in Spoken Dialogue
Details
Knowing when to speak, listen, or yield: evaluating turn-taking across six styles of natural conversation.
“What I cannot create, I do not understand.”— Richard Feynman “I love fools’ experiments. I am always making them.”— Charles Darwin “I have no special talents. I am only passionately curious.”— Albert Einstein “I am among those who think that science has great beauty.”— Marie Curie “We read books to find out who we are.”— Ursula K. Le Guin “The first principle is that you must not fool yourself—and you are the easiest person to fool.”— Richard Feynman “There is no desire more natural than that of knowledge.”— Michel de Montaigne “No way of thinking or doing, however ancient, can be trusted without proof.”— Henry David Thoreau “Nothing is too wonderful to be true, if it be consistent with the laws of nature.”— Michael Faraday “Reading maketh a full man; conference a ready man; and writing an exact man.”— Francis Bacon “The world exists for the education of each man.”— Ralph Waldo Emerson “If I have seen further, it is by standing on the shoulders of giants.”— Isaac Newton “Eagerness for experience, for new and varied contacts, is found where wonder is found.”— John Dewey “Research is formalized curiosity. It is poking and prying with a purpose.”— Zora Neale Hurston “I am like a gambler, & love a wild experiment.”— Charles Darwin “My experience is what I agree to attend to.”— William James “I travel not to go anywhere, but to go.”— Robert Louis Stevenson “I was taught that the way of progress was neither swift nor easy.”— Marie Curie “The value of philosophy is, in fact, to be sought largely in its very uncertainty.”— Bertrand Russell “Travel is fatal to prejudice, bigotry and narrow-mindedness.”— Mark Twain “To punish me for my contempt of authority, Fate has made me an authority myself.”— Albert Einstein “Attention is the rarest and purest form of generosity.”— Simone Weil “One cannot know the rivers till one has seen them at their sources.”— Nan Shepherd “Science progresses best when observations force us to alter our preconceptions.”— Vera Rubin “There are no dangerous thoughts; thinking itself is dangerous.”— Hannah Arendt “One can only show how one came to hold whatever opinion one does hold.”— Virginia Woolf “I like to learn. That’s an art and a science.”— Katherine Johnson “Our knowledge can only be finite, while our ignorance must necessarily be infinite.”— Karl Popper “Our deeds determine us, as much as we determine our deeds.”— George Eliot “I was captured for life by chemistry and by crystals.”— Dorothy Crowfoot Hodgkin “The limits of my language mean the limits of my world.”— Ludwig Wittgenstein “As habit is more dependable than inspiration, continued learning is more dependable than talent.”— Octavia E. Butler “In the field of observation, chance only favors the prepared mind.”— Louis Pasteur “Wonder is the feeling of a philosopher, and philosophy begins in wonder.”— Plato “We can’t question reality directly, only by questioning our conventions, our belief, our orthodoxy, our construction of reality.”— Ursula K. Le Guin “Science and everyday life cannot and should not be separated.”— Rosalind Franklin “All men by nature desire to know.”— Aristotle “Narrative is radical, creating us at the very moment it is being created.”— Toni Morrison “The purpose of computing is insight, not numbers.”— Richard Hamming “Have courage to make use of your own understanding!”— Immanuel Kant “Leave the door open for the unknown, the door into the dark.”— Rebecca Solnit “To doubt everything and to believe everything are two equally convenient solutions; each saves us from thinking.”— Henri Poincaré “He who knows only his own side of the case, knows little of that.”— John Stuart Mill “Not everything that is faced can be changed; but nothing can be changed until it is faced.”— James Baldwin “All models are wrong, but some are useful.”— George E. P. Box “Be a philosopher; but, amidst all your philosophy, be still a man.”— David Hume “You make the path boldly and follow it fearfully.”— Annie Dillard “Science is a way of thinking much more than it is a body of knowledge.”— Carl Sagan “Wisdom comes haphazard to no man.”— Seneca “We may study for ever, and we are never as learned as we would.”— Robert Louis Stevenson
I’m Satya. I train AI models to be useful. I’m currently a researcher at Sesame.
Selected work
Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation
Details
Evaluating factuality, retrieval, and reasoning together, with questions that require information from multiple sources.
D-REX: A Benchmark for Detecting Deceptive Reasoning in Large Language Models
Details
Detecting the gap between apparently benign answers and deceptive reasoning in language models.
Explaining machine learning models with interactive natural language conversations using TalkToModel
Details
Understanding model predictions through a conversation: explanations, data analysis, and follow-up questions in natural language.
Post Hoc Explanations of Language Models Can Improve Language Models
Details
Turning model explanations into natural language rationales that help language models correct their mistakes.
ARES: Adaptive Red-Teaming and End-to-End Repair of Policy-Reward System
Details
Finding safety failures shared by a language model and its reward model, then repairing both.
From Narrow Unlearning to Emergent Misalignment in LLMs
Details
Studying how removing refusals in one domain can weaken safety in other domains.
AutoSUIT Bench - Automated Security UnIt Test Benchmark for LLM Coding
Details
An automatically validated benchmark measuring the security of LLM-generated code across software weakness categories.
Insights from the Inverse: Reconstructing LLM Training Goals Through Inverse Reinforcement Learning
Details
Recovering reward functions from aligned language models to investigate what their training objectives encode.
The Disagreement Problem in Explainable Machine Learning: A Practitioner's Perspective
Details
Measuring how explanation methods disagree, and how practitioners navigate conflicting explanations.
Understanding the Effects of Iterative Prompting on Truthfulness
Details
Studying when repeated prompting reduces truthfulness and calibration, and how prompting strategies can help.
OpenXAI: Towards a Transparent Evaluation of Model Explanations
Details
Tools for comparing the faithfulness, stability, and fairness of model explanations.
BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation
Details
A dataset and evaluation metrics for measuring social biases in open-ended language generation.
ADePT: Auto-encoder based Differentially Private Text Transformation
Details
Exploring text transformations designed to protect privacy while retaining useful semantic information.
TurnBench: A Multi-Domain Benchmark for Turn-Taking Dynamics in Spoken Dialogue
Details
Knowing when to speak, listen, or yield: evaluating turn-taking across six styles of natural conversation.
Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation
Details
Evaluating factuality, retrieval, and reasoning together, with questions that require information from multiple sources.
All papers 14
TurnBench: A Multi-Domain Benchmark for Turn-Taking Dynamics in Spoken Dialogue
Details
Knowing when to speak, listen, or yield: evaluating turn-taking across six styles of natural conversation.
Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation
Details
Evaluating factuality, retrieval, and reasoning together, with questions that require information from multiple sources.
D-REX: A Benchmark for Detecting Deceptive Reasoning in Large Language Models
Details
Detecting the gap between apparently benign answers and deceptive reasoning in language models.
Explaining machine learning models with interactive natural language conversations using TalkToModel
Details
Understanding model predictions through a conversation: explanations, data analysis, and follow-up questions in natural language.
Post Hoc Explanations of Language Models Can Improve Language Models
Details
Turning model explanations into natural language rationales that help language models correct their mistakes.
ARES: Adaptive Red-Teaming and End-to-End Repair of Policy-Reward System
Details
Finding safety failures shared by a language model and its reward model, then repairing both.
From Narrow Unlearning to Emergent Misalignment in LLMs
Details
Studying how removing refusals in one domain can weaken safety in other domains.
AutoSUIT Bench - Automated Security UnIt Test Benchmark for LLM Coding
Details
An automatically validated benchmark measuring the security of LLM-generated code across software weakness categories.
Insights from the Inverse: Reconstructing LLM Training Goals Through Inverse Reinforcement Learning
Details
Recovering reward functions from aligned language models to investigate what their training objectives encode.
The Disagreement Problem in Explainable Machine Learning: A Practitioner's Perspective
Details
Measuring how explanation methods disagree, and how practitioners navigate conflicting explanations.
Understanding the Effects of Iterative Prompting on Truthfulness
Details
Studying when repeated prompting reduces truthfulness and calibration, and how prompting strategies can help.
OpenXAI: Towards a Transparent Evaluation of Model Explanations
Details
Tools for comparing the faithfulness, stability, and fairness of model explanations.
BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation
Details
A dataset and evaluation metrics for measuring social biases in open-ended language generation.
ADePT: Auto-encoder based Differentially Private Text Transformation
Details
Exploring text transformations designed to protect privacy while retaining useful semantic information.
Background
I completed my PhD at Harvard’s School of Engineering and Applied Sciences, advised by Hima Lakkaraju and Finale Doshi-Velez. My research focused on understanding and improving the trustworthy aspects of generative models. I also collaborated with Sameer Singh at UC Irvine and Steven Wu at Carnegie Mellon.
Previously, I worked at Google DeepMind and Meta FAIR, and led research initiatives across AWS, Alexa, and Amazon Search. At Carnegie Mellon, I studied conversational agents and reinforcement learning.
PhD · Computer ScienceHarvard University
School of Engineering and Applied Sciences
Master’s degreeCarnegie Mellon University
School of Computer Science
Doctoral thesis: From Understanding to Improving Artificial Intelligence: New Frontiers in Machine Learning Explanations.
View my CVAcademic service & community
- Co-President · Harvard SEAS Graduate Council, 2022
- Program Committee · ICML Workshop on Human-Machine Collaboration and Teaming, 2022
- Reviewer · NeurIPS Datasets and Benchmarks, 2021–2022; NeurIPS Main Conference, 2022; AAAI, 2021
- Organizing Committee · Workshop on Trustworthy NLP at NAACL, 2021
- Student Volunteer Award · ACL, 2022
- Volunteer · ACM FAccT, 2022
News
I’ve joined Sesame, where I am building the best voice model.
Introducing TurnBench: a benchmark, dataset, and leaderboard for turn-taking in spoken dialogue.
Released D-REX, a benchmark for detecting deceptive reasoning in language models.
Harvard PhD thesis: From Understanding to Improving Artificial Intelligence: New Frontiers in Machine Learning Explanations.
Introduced FRAMES, a benchmark for factuality, retrieval, and reasoning. Published at NAACL 2025.
Earlier news
Teaching Fellow for COMPSCI 236R: Topics at the Interface between Computer Science and Economics.
Amazon Science highlighted my contributions to trustworthy machine learning at Alexa AI.
Presented The Disagreement Problem at the TRAIT Workshop at CHI. Two papers on algorithmic fairness accepted at ACL: paper one and paper two.
Released The Disagreement Problem, later published in TMLR. Covered by Fortune and Analytics India Magazine. Received the Nicole A. Chen and Karina A. Chen Graduate Student Research Fellowship.
Joined Harvard SEAS as a Computer Science PhD student.
Does Robustness Improve Fairness? accepted at Findings of ACL 2021.
VentureBeat covered BOLD, our benchmark for measuring bias in open-ended language generation.
ADePT, our work on private text transformation, accepted at EACL 2021.
Announced BOLD, a fairness benchmark for open-ended language generation, presented at FAccT 2021.