Can AI count how often a theme appears?

No, and this is the single most important rule in this article: **the AI labels, the spreadsheet counts.** Language models produce plausible numbers rather than correct ones, and the frequency is exactly the figure that ends up on the management slide. Let the model assign one category per answer, export the labels to a table, and count with a pivot. If you ask the model directly how many people mentioned communication, you will get a confident number nobody can reproduce — and if you run the same prompt again you may well get a different one.

How reliable is AI sentiment analysis in German?

Good enough for a trend, not good enough for a verdict on an individual answer. One survey vendor reports about 78 percent perfect agreement on sentiment — which also means roughly one in five answers is classified wrongly. Sarcasm is much worse: research on language nuances found accuracy around 30 percent on sarcastic content, and removing the sarcasm improved sentiment accuracy by up to 21 percent. German makes this harder through understatement and litotes — "nicht schlecht" is positive, "an sich ganz nett, aber..." is negative, and "läuft ja super hier" may be the opposite of what it says. Always allow the model an explicit "unklar" option, and never let a sentiment label alone trigger a decision about a person or a team.

Why do the categories change when I analyse in several batches?

Because you let the model invent them again each time. If you paste 500 answers in five chunks without supplying a fixed codebook, the model creates fresh categories in batch three that did not exist in batch one — and your frequencies become incomparable. The fix is to freeze the codebook first: derive six to ten categories from a sample of 50 to 80 answers, write a one-sentence definition and two anchor examples for each, and then supply that same codebook with every single batch while forbidding new categories. Also keep batches small, around 40 to 60 answers, because models attend less reliably to material in the middle of very long inputs.

May I paste employee survey answers into ChatGPT?

Not into a consumer account, and not without thinking about what is in the text. Free and personal paid ChatGPT tiers are consumer products without a data processing agreement, and content may be used for training by default — that makes them unsuitable for third-party data. Business, Enterprise and API tiers offer a processing agreement and no default training. Beyond the contract, free text from an employee survey is regularly personal data even when the survey was called anonymous, because people write names, projects and events: in an eight-person team, "ich habe letzte Woche als Projektleiter den Rollout verantwortet" identifies someone. The safest route is to pseudonymise before upload, locally, and keep a minimum group size of five before releasing any group-level result.

Does the works council have to be involved?

For employee surveys, generally yes. Section 87(1) no. 6 BetrVG grants co-determination over technical systems that are objectively suitable for monitoring behaviour or performance — and the suitability is what triggers the right, not whether you intend to monitor anyone. AI evaluation makes this more sensitive rather than less, because sentiment analysis that profiles groups is exactly the kind of processing the provision has in mind. The workable path is a works agreement covering purpose, anonymity thresholds, storage location, access rights and deletion periods, with monitoring use explicitly excluded — and all technical functions including the AI analysis disclosed. Customer feedback is a different matter and does not touch BetrVG.

Analyse Surveys With AI: 6 Prompts + Failure Modes

The Method in 8 Steps

Short answer: freeze a codebook first, then code in small batches, then count outside the AI. Everything that goes wrong with AI survey analysis goes wrong because one of those three steps was skipped. The approach mirrors thematic analysis from qualitative research — the difference is only that the coding is done by a model that needs its rules supplied every single time.

Prepare the data

One answer per row with a running ID (A001, A002). Remove empty rows and non-answers. Keep the team or segment column separate — if you hand it over, the model starts guessing along group lines.

Pseudonymise

Replace names, projects, locations and dates with placeholders before anything leaves your system. Prompt 1 does this, but run it where the data is allowed to be.

Build the codebook from a sample

50 to 80 answers, six to ten categories, each with a definition, a boundary and two anchor examples. This is the step everyone skips and the reason their categories drift.

Freeze it

From here the codebook does not change. It gets supplied with every batch, and new categories are forbidden.

Code in batches of 40 to 60

Not 500 at once. Output as CSV with the ID, never as prose. Allow the model to answer nicht_zuordenbar — forced choices produce invented ones.

Sentiment in a separate pass

Asking for theme and sentiment together makes both worse. Separate call, with an explicit unklar option.

Count in a spreadsheet

Pivot over the labels. The AI never produces the frequency — this is the rule that keeps your management summary defensible.

Validate 10 percent yourself

Code a random tenth by hand and compare. Above roughly 15 percent disagreement, sharpen the codebook rather than switching model. Automated annotation requires human validation — that is the documented consensus, not a nicety.

The 6 Prompts

Copy these as they are. Set the model to a low temperature for coding tasks, note which model version you used, and keep it the same across all batches — changing the model mid-analysis means you are measuring the model change rather than the responses.

Du bist ein Datenschutz-Assistent. Deine einzige Aufgabe ist Pseudonymisierung.

Ersetze in JEDER Antwort:
- Personennamen → [PERSON_1], [PERSON_2] (gleicher Name = gleiche Nummer)
- Führungskräfte mit Namensbezug → [FÜHRUNGSKRAFT]
- Team-, Abteilungs-, Standort-, Firmennamen → [TEAM], [STANDORT], [FIRMA]
- Projekt- und Produktnamen → [PROJEKT]
- Konkrete Daten → [DATUM]

Regeln:
- Ändere sonst KEIN Wort. Keine Umformulierung, keine Korrektur von Tippfehlern.
- Gib jede Zeile zurück, auch unveränderte.
- Wenn eine Antwort trotz Ersetzung eine Person identifizierbar macht, markiere sie am Zeilenende mit RESTRISIKO.

Ausgabe: ID | anonymisierte Antwort | RESTRISIKO (falls zutreffend)

Antworten:
[HIER EINFÜGEN]

Du bist erfahren in qualitativer Inhaltsanalyse.

Kontext: Befragung mit [ANZAHL] Teilnehmenden. Die Frage lautete wörtlich: "[FRAGE]"

Aufgabe: Entwickle aus den folgenden Antworten ein Codebuch. Ordne noch KEINE einzelnen Antworten zu.

Anforderungen:
- 6 bis 10 Kategorien. Nicht mehr. Aggregiere, wenn du mehr Themen siehst.
- Die Kategorien müssen trennscharf sein.
- Sie müssen aus DIESEN Daten stammen, nicht aus allgemeinem Vorwissen.
- Keine Kategorie darf so allgemein sein, dass mehr als 40 Prozent der Antworten hineinpassen.

Liefere je Kategorie:
1. Kurzname (max. 3 Wörter)
2. Definition in einem Satz
3. Abgrenzung: "Nicht hierher gehört: ..."
4. Zwei Ankerbeispiele, WÖRTLICH aus den Daten, mit ID

Ergänze die Pflichtkategorie "nicht_zuordenbar".
Nenne am Ende Themen, die in keine Kategorie passten.

Antworten:
[50-80 ANTWORTEN EINFÜGEN]

Du codierst Antworten nach einem FESTEN Codebuch. Du darfst es nicht erweitern, umbenennen oder umdeuten.

CODEBUCH (verbindlich):
[CODEBUCH AUS PROMPT 2 EINFÜGEN]

Regeln:
- Genau EINE Hauptkategorie je Antwort.
- Optional eine Nebenkategorie, wenn ein zweites Thema klar erkennbar ist.
- Erfinde KEINE neue Kategorie. Passt nichts: "nicht_zuordenbar".
- Bei Unsicherheit: "nicht_zuordenbar" plus Grund in der Spalte Notiz. Unsicherheit ist ein gültiges Ergebnis, Raten nicht.
- Bearbeite die Antworten in der gegebenen Reihenfolge, überspringe keine.
- Gib exakt so viele Zeilen zurück, wie du Antworten bekommen hast.

Ausgabe als CSV mit Kopfzeile, ohne Einleitung:
ID;Hauptkategorie;Nebenkategorie;Konfidenz(hoch|mittel|niedrig);Notiz

Antworten:
[BLOCK VON MAX. 60 EINFÜGEN]

Du bewertest die Grundstimmung von Freitextantworten. Eigener Durchgang, unabhängig vom Thema.

Regeln:
- "unklar" ist ausdrücklich erwünscht bei sehr kurzen Antworten ohne klare Wertung, reinen Sachaussagen, erkennbarer Ironie und gemischten Aussagen ohne Übergewicht.
- Achte auf deutsche Untertreibung: "nicht schlecht" = eher_positiv, "an sich okay, aber ..." = eher_negativ, "läuft ja super hier" im ironischen Kontext = unklar, NICHT positiv.
- Bewerte die Stimmung der schreibenden Person, nicht das Thema. Eine höflich formulierte Beschwerde ist negativ.
- Bewerte NICHT, wie wichtig die Aussage ist.

Ausgabe als CSV, ohne Einleitung:
ID;Sentiment;Ironie_vermutet(ja|nein);Belegwort(wörtlich aus der Antwort)

Antworten:
[BLOCK EINFÜGEN]

Du wählst Belegzitate für einen Ergebnisbericht aus.

ABSOLUTE REGEL: Jedes Zitat muss zeichengenau im Ausgangstext stehen. Nicht umformulieren, nicht glätten, keine Tippfehler korrigieren, nichts zusammenziehen. Wenn du versucht bist zu formulieren, nimm ein anderes Zitat.

Aufgabe: Wähle zur Kategorie "[KATEGORIE]" bis zu 4 Zitate.
Kriterien in dieser Reihenfolge:
1. typisch für die Kategorie, nicht das dramatischste
2. ohne Zusatzkontext verständlich
3. enthält keine identifizierenden Details
4. eines soll eine ABWEICHENDE Sicht innerhalb der Kategorie zeigen, falls vorhanden

Je Zitat: ID, Zitat wörtlich, die ersten 4 Wörter noch einmal separat zur Prüfung, und in einem Satz warum du es gewählt hast.

Findest du weniger als 4 geeignete, gib weniger zurück und schreibe "Nur X geeignete Zitate gefunden". Fülle NICHT auf.

Quelltexte:
[ANTWORTEN DER KATEGORIE EINFÜGEN]

Du suchst gezielt das, was in einer Zusammenfassung untergehen würde.

Finde in den folgenden Antworten:
A) Aussagen, die dem Mehrheitsbild widersprechen, auch wenn sie nur ein- oder zweimal vorkommen
B) Themen von weniger als 5 Prozent der Antwortenden, die inhaltlich schwer wiegen (Konflikte, Überlastung, Fehlverhalten, Sicherheit, rechtliche Themen)
C) innere Widersprüche: Antwortgruppen, die sich zum selben Sachverhalt widersprechen

Regeln:
- Fasse NICHT zusammen und glätte NICHT. Zitiere wörtlich mit ID.
- Bewerte nicht, ob die Aussage stimmt. Du sammelst nur.
- Sortiere NICHT nach Häufigkeit. Eine einzelne Nennung kann an erster Stelle stehen.
- Findest du zu einer Rubrik nichts, schreibe "keine gefunden". Erfinde nichts.

Ausgabe:
A) ID – wörtliches Zitat – wem es widerspricht
B) ID – wörtliches Zitat – warum relevant
C) Sachverhalt – ID und Zitat je Seite

Antworten:
[ALLE ANTWORTEN EINFÜGEN]

What the AI Gets Wrong

No vendor page lists these, because every vendor page is selling the analysis module. These are the failure modes you will actually meet, in the order of how much damage they do.

Failure mode	What happens	Countermeasure
Invented quotes	A plausible, lightly reworded or wholly fabricated quote lands on slide 3 and nobody checks it	Verbatim constraint + ID + search the raw data for the first four words
Divergence from the source	Thematic summaries can drift from the true substance of the responses — documented for exactly this task	Validate 10 percent by hand; treat the summary as a draft
Minority views disappear	Models favour majority-aligned content; the four people hinting at a conflict vanish into "wish for better communication"	Run prompt 6 as a separate pass, every time
Sarcasm and understatement	Accuracy on sarcastic content drops to around 30 percent; German litotes reverses the sign	Explicit unklar option; never act on a single sentiment label
Category drift	Batch three invents categories batch one never had, and the counts stop being comparable	Freeze the codebook, resend it every batch, forbid new categories
Lost in the middle	Answer 250 of 500 gets less attention than answer 1 — positional bias is well documented	Small batches; reverse the order once as a control
Different result on rerun	Output varies between runs even with identical prompt and temperature zero	Classify critical items several times; pin the model version

The AI labels. The spreadsheet counts. A language model produces plausible numbers, not correct ones — and the frequency is precisely the figure that ends up in front of management. Let the model assign categories, then count the labels yourself. That single separation prevents most of the damage on this page.

GDPR and Free Text

Free text from a survey is regularly personal data even when the survey was announced as anonymous, because people write names, projects and events into it. In an eight-person team, "ich habe letzte Woche als Projektleiter den Rollout verantwortet" identifies someone as surely as a name would. Re-identification usually comes from small groups, specific events, recurring writing style and mentioned names — not from a missing anonymisation checkbox.

Three practical consequences. First, a processing agreement is required as soon as personal data reaches an external model — consumer ChatGPT tiers do not offer one and may use content for training by default, which rules them out for third-party data; business and API tiers do offer one. Second, pseudonymise before upload and do it where the data is already allowed to be — letting the model anonymise raw data means you have already transferred it. Third, hold a minimum group size of five before releasing any group-level result; larger organisations and those with a works council commonly use ten. Tell respondents that AI is used for the analysis — that is a transparency obligation, and it also raises the response rate.

Doing It With teamo

Everything above describes doing the analysis by hand with a general-purpose model, which works and costs you an afternoon per survey. The alternative is to collect the answers somewhere that already does the coding step under the same rules — with EU hosting, a processing agreement in place, and the minimum-group-size threshold enforced in the software rather than in a policy document.

There is also a way to avoid part of the problem entirely. Most of the difficulty in this article comes from open answers that are too vague to categorise — "die Kommunikation könnte besser sein" cannot be coded into anything actionable no matter how good your prompt is. An AI interview that asks a follow-up question while the person is still answering produces text that is specific in the first place, which removes more analysis work than any prompt engineering does.

Collect and analyse in one place — 14 days free

Answers come in already specific because the AI asks follow-up questions, themes and quotes are summarised for you, and the group-size threshold is enforced by the software. EU-hosted. Free for 14 days, no credit card.

Try It Free

Analysing Surveys With AI: Method, Prompts and What the AI Gets Wrong