Synergised Consulting
Technical asset

Colgate's Synthetic Consumers Hit 90% of Human Survey Reliability. The Interesting Part Is How They Ask

Synergised Consulting Ltd

6 min read
Title card: Synthetic consumers now match real survey results, Colgate-Palmolive and PyMC Labs purchase-intent study, 90 percent of human test-retest reliability

Last Updated: 22 November 2026

A study by Colgate-Palmolive and PyMC Labs shows that AI-simulated survey respondents, asked the right way, reproduce real human purchase-intent results at 90% of the reliability you would get from running the same survey on humans twice. The finding comes from 57 real consumer surveys with 9,300 human responses, and the method that made it work is as instructive as the result: asking a model directly for a score produces a useless answer, while asking it to describe its reaction and then scoring the description works. If you have ever wondered whether an AI could give you a cheap first read on a product idea before you pay for a full panel, this study is the strongest evidence yet that the answer is a qualified yes.

What Synthetic Consumers Are, and What SSR Actually Does

Synthetic consumers are language models prompted to act as survey respondents: each one is given a demographic profile and a product concept, then asked for its purchase intent on a five-point scale. The Colgate-Palmolive and PyMC Labs study [1] tested this against a substantial real baseline: 57 personal care product concept surveys run through the company's consumer research platform, covering 150 to 400 unique US participants each and 9,300 human responses in total. Human purchase intent in this dataset clustered heavily positive, averaging 4.0 on the five-point scale, which matters because it sets a narrow target for anything trying to imitate it.

The method at the centre of the paper is semantic similarity rating, or SSR. Instead of asking the model for a number, it asks the model to write a short free-text reaction to the concept, then embeds that text and compares it against reference statements written for each point of the scale. The comparison produces a probability distribution across all five points rather than a single number, which preserves uncertainty the way a real panel does. No fine-tuning was involved: the study used off-the-shelf models with carefully written prompts and reference statements.

Why You Cannot Just Ask for a Number

The study's cleanest result is the failure case. When models were asked directly for a 1-to-5 rating, they regressed to the middle: they almost always answered "3", rarely volunteered "1" or "5", and produced response distributions that barely resembled the real ones. On the study's distributional similarity measure, direct rating scored 0.26 for GPT-4o and 0.39 for Gemini-2.0-flash, against a perfect score of 1.0 [1]. The authors noted that nudging the models harder to spread their answers made the distributions look better but destroyed the correlation with real product rankings, which is the signal a concept test exists to produce.

The shape of that failure is worth recognising, because it is not really about surveys. A model asked for a compressed judgement tends to hand back the safest point on the scale, and the safest point is the middle. The information you wanted was in the extremes, and the format of the question quietly deleted it.

What the Method Got Right, and Where the Authors Say It Stops

With SSR, distributional similarity rose to 0.88 for GPT-4o, and correlation attainment reached 90% of the human test-retest ceiling across the 57 surveys, according to the study [1]. That ceiling matters:

The Benchmark That Makes 90% Mean Something

Because human purchase-intent means sat in a narrow band, the researchers first measured how well two halves of the real human data correlate with each other, then judged the synthetic panel against what a repeated human study could actually achieve. Against that honest benchmark, 90% is a strong result. The synthetic respondents also produced something human surveys rarely capture at scale: written rationales explaining each rating, which can be mined for objections and themes rather than just counted.

The Control Result Anyone Applying This Should Not Skip

There is a second result that does real work for anyone thinking about applying this. When the researchers removed demographic information from the personas entirely, the response distributions still looked realistic, scoring 0.91 similarity, but the correlation with real product rankings collapsed to 50%, against 92% with demographic personas present [1]. Realistic-looking output and useful output are different properties. The demographics were not decoration; they carried the signal. Anyone wiring up a quick synthetic panel without persona structure would get plausible numbers that rank products no better than chance. Plausible is not the same as valid, and this study shows the gap between the two can be invisible in the output itself.

The limitations section is equally worth reading, because the authors are unusually direct about them. The reference statements used for the Likert mapping were manually optimised for these 57 surveys, and the study does not claim they transfer to other survey designs. Patterns by gender, region and ethnicity were not consistently replicated, so the authors warn against treating synthetic panels as a reliable proxy for subgroup analysis. And validity is bounded by training data: the method worked in oral care because public discussion of those products is abundant, and the paper is explicit that in domains with sparse background coverage, SSR will not conjure valid consumer preferences. Independent coverage of the study made the same scope point, noting that testing covered one product category from a single corporation and that broader validation is still needed [2].

What an Owner Can Usefully Take From It

The general lesson travels further than market research: the way you elicit an answer from a language model determines whether you get its real knowledge or its safest hedge. In discovery workshops, we see the same shape from the other side. Owners rarely arrive with a list of what to automate; the list gets built in the conversation, walking through a normal week task by task, because the useful specifics only surface when someone is asked to describe rather than to summarise. This study is that observation, run at scale with numbers attached.

For a founder validating a new product concept, the practical reading is this: a synthetic panel is now a defensible cheap first pass for idea screening in a well-covered domain, not a replacement for a real panel. If you ever run one, treat it the way the study's own structure implies it should be treated. Write down which reference statements and which embedding model you used, which domain you tested in, and where the method's authors say it stops; check any decision that matters against a small real sample before committing spend. A synthetic study with a written calibration record is inspectable; a synthetic study without one is just a confident-sounding number.

Sources

  1. [1] Maier, Pappas et al., "LLMs Reproduce Human Purchase Intent via Semantic Similarity Elicitation of Likert Ratings", arXiv:2510.08338:
  2. [2] PPC Land, "LLMs achieve 90% accuracy in replicating consumer purchase intent" (qualitative framing on scope limits only, no figures carried):