How we measure Survey Coding quality
Last measured: 16 November 2026 · model s1-fast-2 (Decision Models API, EU tier) · Beta
Survey Coding asks the model one yes/no question per answer and code ("Does the answer mention or express this theme?") and gets back a probability. You decide what happens with uncertain answers: they are flagged for your review.
What we measured
We used a held-out test set the model and our settings never saw during tuning:
| Data set | Answers | Codes | Micro-F1 of model suggestions (after calibration on 40 answers) | Answers flagged for review | Micro-F1 after you review the flagged answers |
|---|---|---|---|---|---|
| Synthetic: NPS reasons (SaaS), EN | 350 | 12 | 0.95 | 22 % | 0.98 |
| Synthetic: NPS reasons (online shop), DE | 350 | 12 | 0.98 | 7 % | 0.99 |
| Synthetic: employee survey, EN | 350 | 12 | 0.95 | 26 % | 0.99 |
| Synthetic: employee survey, DE | 350 | 12 | 0.97 | 13 % | 0.99 |
| MAMS restaurant reviews (human-labelled), EN | 400 | 8 | 0.67 | 86 % | 0.97 |
Micro-F1 combines precision (suggested codes that are right) and recall (right codes that were suggested) over all code decisions; 1.00 would mean every decision matches the reference labels.
Calibration on 40 answers is the step the app offers for projects with 150+ answers: you review 40 random answers, the app fits its thresholds to your decisions, then codes the rest. In the measurement, the reference labels played the part of the reviewer; we report the mean over 20 random samples of 40.
How the data sets were made — please read before you compare
- Synthetic sets (4): answers to typical survey questions (NPS reason, employee survey; English and German) written by a language model to express chosen codes, with short and long answers, typos, sarcasm and negations. We kept an answer only if an independent second model assigned exactly the same codes. The labels are therefore not human-coded, and the filter removes ambiguous answers: real survey data is harder.
- MAMS (public, human-labelled): restaurant review sentences with 8 aspect categories, each sentence has two or more aspects, including a catch-all "miscellaneous" category (Jiang et al., 2019, Apache-2.0). On this harder data the app flags most answers for review.
- Your customer uploads are never used for measurements, prompt tuning or training.
What this means for you
Your data can differ. Use the free tier (up to 500 answers) and the calibration step to see how the model does on your own codebook before you rely on it. All model suggestions are AI-generated; you review and own the final codes.