PRD: Loupe v1

What this is for: Define what Loupe must do, for whom, at what priority, and how launch is judged, before dashboard and pipeline work is finalized. Date: 2026-09-20 Status: Final Version: 0.1

Problem

A person who owns an AI assistant feature cannot answer who uses it, how intensely, for what, or where it fails, without an engineer writing a one-off script. Priya is the product manager for the AI assistant inside a customer-support tool, and about 40,000 conversations a week flow through it. Her VP asks three questions on a Monday: is usage growing or is it the same power users every week, what are people using it for, and support tickets say the assistant "doesn't get it" so where is it failing. Today she asks a data engineer for a SQL pull, waits three days, gets a spreadsheet of 2,000 conversations, and reads a few hundred by hand over a weekend to form a gut impression that nobody else can check. A month later the same questions come back and she repeats the whole cycle.

This recreates a pattern documented across the industry: most teams shipping an AI feature either run no evaluation at all or use ad-hoc scripts that only an engineer can operate, so the PM who is accountable for the feature's quality and adoption cannot act on her own. Loupe closes that gap with a metrics pipeline and dashboard that a PM or analyst runs and reads without engineering help, demonstrated end to end on WildChat-4.8M as a stand-in for a team's own production logs.

Goals

• G1. Ship a complete, public, ordered set of senior-PM artifacts for one product, each dated and versioned in the repo.
• G2. Ship a working, publicly hosted dashboard backed by metrics computed over all 3.2M conversations, reproducible from one command.
• G3. Publish a trends report with at least five findings a product owner would act on, each traceable to a metric definition and a query.
• G4. Complete six or more user interviews and a synthesis document; ship with whatever count lands, plus a plan for the rest.
• G5. Handle the sensitive-data dimension explicitly and well, because the target roles ask for it.
• G6. Meet the v1 launch criteria in Section 6.3, including a five-participant usability test and validation of every friction proxy against hand labels.
• G7. Pass all five launch checks in "Launch criteria" below.

Non-goals

• Not a general-purpose eval framework. No metric authoring UI, no CI integration, no LLM-as-judge scoring of every response.
• Not a live ingestion product. No connectors to production logging systems in v1.
• Not a research paper. No claims about ChatGPT's user base at large; claims are about this dataset's population, stated as such.
• Not toxicity or safety analytics in v1. The public dataset has toxic conversations removed, so safety rates cannot be computed honestly from it. See Roadmap → Later.
• No re-hosting of the raw dataset. Hugging Face remains the source of truth.
• No changes to shankard.com. Loupe ships its own site; embedding or linking it from the personal site is outside this project.

Personas

Priya, PM at a mid-size SaaS company. Context: owns the AI assistant inside a customer-support product; about 40,000 conversations a week; has a data engineer but no dedicated analyst, and every request to that engineer competes with other work. Job: answer her VP's volume, intent, and failure questions on a weekly or ad-hoc basis without filing a ticket. Good looks like: she opens the dashboard on a Monday, gets an answer with a caveat attached inside two minutes, and cites the number in the VP meeting without anyone disputing it.

Dev, the analyst who owns the SQL. Context: the one person on the team who can write the queries the PM's questions actually require; treated as a bottleneck because every product question becomes an engineering ticket. Job: stop being paged for questions that do not need a new query, and trust that the numbers the PM pulls herself are correct. Good looks like: Dev maps the team's logs to Loupe's schema once, then gets pulled in only for genuinely new questions, and can point to the Query view's SQL when someone doubts a number.

Sam, the founder running a chatbot with no analyst on staff. Context: ships a support or tutoring assistant, has no one dedicated to data, and currently reads a handful of transcripts by hand on weekends to get a gut feel for quality. Job: get a systematic read on volume, intent, and failure without hiring an analyst or writing SQL. Good looks like: Sam maps logs to the schema, opens the dashboard, and replaces the weekend transcript-reading habit with a five-minute weekly look at the same five views.

User stories

1. As Priya, I want a volume view sliced by model, country, and language, so that I can tell my VP whether usage is growing and where it is coming from.
2. As Priya, I want an intensity view with return rate and top-10%-share, so that I can tell whether growth is broad or concentrated in a shrinking core of power users.
3. As Sam, I want an intent view built from a fixed taxonomy, so that I know what people are actually trying to do without reading transcripts myself.
4. As Priya, I want a friction view with proxies for repeated requests, corrections, one-turn abandonment, and refusals, so that I can tell engineering exactly which failure mode to fix first.
5. As any owner, I want every number I export to carry its population caveat and its pseudo-user caveat, so that I can defend the number when someone on my team questions it.
6. As Dev, I want a Query view that shows the exact SQL behind any number, so that I never have to re-derive a query from scratch when someone doubts a dashboard figure.
7. As any owner, I want the entire aggregate set to regenerate from one committed command against the same input data, so that I can prove a number is reproducible rather than a one-off pull.
8. As a hiring PM reviewing this project, I want the dashboard and every underlying aggregate to contain no row-level transcript content, so that I can trust the product handles sensitive user data the way the target roles require.

Requirements

Labeling (Stage 3a) ran on 2026-09-25 through the UCSD TritonAI gateway (claude-sonnet-5): 9,829 of 10,000 sampled conversations labeled. Classification (Stage 3b) ran on 2026-09-26 with taxonomy v2 and passed the restated gate (78.6% held-out against 78.4%); intent_coverage is full and R1 and R3 are met for the intent view, with the reliability figures shown on the view itself.

Success metrics

These are the metrics for Loupe itself, distinct from the analysis metrics Loupe computes about an assistant (defined in 03-metrics-framework.md).

North star: weekly answered questions. A session in which a user reaches a view and exports, shares, or copies a number with its caveat attached. Usage alone does not count; the user has to take something away.

Metrics tree, for Loupe in market

Guardrails

Any of these blocks a release regardless of the metrics above.

• Zero row-level transcript content reachable from the dashboard.
• Minimum-cell suppression applied on every geography and language slice.
• Intent classifier agreement with LLM labels at 85% or higher on the held-out sample. Amended 2026-09-26: an inter-rater study (two independent labeling models on 348 conversations) measured 84% agreement on the ten-class taxonomy and 87% on the revised seven-class taxonomy. A classifier cannot be expected to exceed the agreement of the raters that produced its labels, so the gate is restated as: held-out accuracy at or above 90% of measured inter-rater agreement on the current taxonomy (78.4% for taxonomy v2). The absolute 85% figure remains the fallback when no rater study exists. The taxonomy revision and the study are recorded in artifact 03 and the PRD decision log.
• Friction proxy precision at 70% or higher against 300 hand-labeled conversations (Section 13).
• Pipeline cost per million conversations under the cap set in the PRD before the classifier stage runs.
• Aggregates never older than one week from the latest log.

Launch criteria

Loupe has one "team" (the WildChat demonstration) and no in-market users at launch. The in-market metrics above are therefore unmeasured at launch, and the PRD says so. The launch criteria below are proxies, each mapped to the metric it predicts. Instrumenting the real metrics is a roadmap item.

Plus: docs check and report check pass.

All five checks are unmeasured at this version, per the reframed research method in 02-user-research.md (no interviews were conducted in the project window). They remain the defined bar for launch; the pending recruitment plan is carried in 02-user-research.md → "Gaps and next steps."

Privacy and sensitive data

What the dataset authors already did. WildChat-4.8M is de-identified by its authors before release: Presidio and regex-based PII detection, manual review of a sample, and TruffleHog scanning for leaked secrets. Toxic conversations were removed from the public build entirely, which is why Loupe defers safety and toxicity metrics to a future dataset build (see Roadmap → Later, and the decision log below).

What Loupe adds on top of that. Loupe never displays raw transcript content on the public dashboard; only aggregates leave the pipeline, and the committed aggregates/*.parquet files carry no conv_id-to-text mapping. Minimum-cell suppression (min_cell = 20) applies to every geography and language slice, and suppressed or excluded rows are published as an explicit residual row rather than silently dropped, so a reader cannot infer that a small cell was hidden versus simply absent. The intent-classification sample, the one artifact in the pipeline that touches conversation content directly, is sent to the labeling API under the project's own credential and is never committed to the repo; the local cache of that sample (intent_text, capped at 2000 characters per conversation, kept only so classification runs in one pass) lives outside version control. The trends report and dashboard quote no conversation verbatim.

How a removal request is honored. The repo README documents the procedure: if the dataset authors forward a data-removal request, the affected shard is removed from the local cache and the full pipeline (ingest through metrics) is re-run against the updated corpus, regenerating every aggregate from source. Because Loupe holds no independent copy of raw conversation content, honoring a removal request is a re-run, not a manual redaction.

Why hashed IP is treated as personal data anyway. The pseudo-user key (loupe/text.py:pseudo_user) is a hash of IP address, user agent, and accept-language, and the dataset's own IP field is already hashed before Loupe ever sees it. A hash is not the same as anonymization. The same input reliably produces the same hash, so anyone who can compute the same hash function over a candidate IP, user agent, and accept-language tuple can test whether a specific real person's traffic is in the dataset. The combination of IP-adjacent signal, browser fingerprint, and language is itself a quasi-identifier, even without decrypting anything. Loupe therefore treats every pseudo-user key as pseudonymous personal data, not anonymous data: it is never shown or exported at row level, it appears only inside aggregate counts, and it is subject to the same minimum-cell suppression as any other identifying slice.

Open questions

• Which charting library the dashboard uses. Resolved in 06-dashboard-design.md.
• Final boundaries of the ten-class intent taxonomy once labeling has actually run at scale; the taxonomy in 03-metrics-framework.md was finalized by hand-reading 200 conversations, not by running the classifier against real held-out accuracy numbers.
• Whether to host aggregates on MotherDuck in addition to static parquet files (R10, P2), and if so, on what schedule.
• Whether intent classification should ship sample-only (the ~10,000-conversation labeled slice, with that scope stated) if the classifier fails the 85% held-out accuracy gate, rather than extending to the full corpus.
• All six assumptions in the assumptions register (02-user-research.md → "Assumptions register") remain open until primary research runs: how owners answer usage questions today, how they judge quality, whether a caveat increases trust, their decision cadence, whether intent is the first metric family they want, and whether the friction proxies track real user outcomes.

Decision log

After interviews complete, this document moves to Version: 1.0. Each PRD change made because of interview findings gets its own row here, tagged with the participant ids that motivated it, mirrored under "What changed in the PRD" in 02-user-research.md.