3,199,860 conversationsApr 2023 – Jul 2025 · summary counts onlya case study by Shankar Deenadayalan · resume
for teams shipping
GenAI assistants

See who your
assistant serves

→ not another dashboard.
a loupe.

The dashboard said the cheap AI model failed 3× as often. For the same people, it didn't. Loupe shows what's really going wrong.

The problemAI teams pick models and judge changes from dashboards that blame the model for who uses it.
What Loupe doesReads your chat logs, separates real model gaps from who-uses-what, and hands you a fix list.
What's real3.2 million real conversations. Summary counts only, every number re-checked on each update.

A short animation. On a usage dashboard, people correct the cheaper model 3.2 times as often as the bigger one, so the dashboard says drop it. But the two models weren't used by the same kind of people, so the gap mixes the model with who picked it. For the 622 people who used both, the rates are about the same, 11.6% and 11.6%. Loupe runs this check on your own logs and hands you the results: a verdict on your last change, your failures as tests, and a backlog.

Loupe

See who your assistant serves.

On the dashboard, people corrected the cheap model 3.2× as often. Among the 622 people who used both models, they corrected both about equally often.

Open the live dashboard Try it on your logs

dots stand for people · the WildChat numbers are real

the four questions, answered from the logs

    and what it says to do

    01Don't pick a model from usage logs aloneThe logs suggested that sending each kind of request to a different model would make far fewer conversations end after one short reply. Comparing the same people on both models showed almost none of that difference came from the model. The write-up keeps both results and the correction. 02Test an assistant on what people really askHow often people ask for each kind of help, published so a team can weight its own test set the same way, with a scorer that names the kinds of requests the set never tests.
    Open the dashboard Try it on your own logs Read the case study

    Data: WildChat-4.8M (Zhao et al., ICLR 2024; Deng et al., EMNLP 2024 Demos), licensed ODC-By 1.0. Aggregates only; no transcript content is served. The chatbot behind these logs was run by researchers, not ChatGPT itself.

    A product case study by Shankar Deenadayalan · resume · source. This site uses PostHog to understand how it is used; the WildChat data itself is never tracked at row level.