Give a model a few hundred reviews and ask "what do people praise and complain about?" and in a minute you get a report that reads well. It has two defects that are easy to miss, because both still look right:
- The quotes aren't quotes. Models like to tidy a comment up and then put quote marks around the tidy version. The reviewer wrote 「对话部份错误较多」 ("quite a few mistakes in the dialogue"), and the report says 「对话错误很多」 ("lots of dialogue errors"). It means about the same, but nobody actually wrote it.
- The numbers are guesses. "Viewers of 5 dramas complain about plot holes." That 5 is often the model's impression. Run it again and you get 4, or 6.
The fix comes down to one rule: the model only does what it is good at, which is putting comments that mean the same thing into a group. Code does the counting and checks every quote.
The examples come from public viewer reviews on a Chinese short-drama site (48 dramas, 432 reviews, read on 2026-11-15). I ran the grouping step through a link-reading tool I build, LinkDigest, but nothing here depends on it. It works the same with any model.
Step 1: give every comment an id
A quote has to point back to one specific comment, so number them first:
reviews = [] # each: {"id", "drama", "text"}
for drama in dramas:
for i, text in enumerate(drama["reviews"]):
reviews.append({"id": f'{drama["id"]}#{i}',
"drama": drama["id"], "text": text})
by_id = {r["id"]: r for r in reviews}
Step 2: ask for groups, never for numbers
The prompt asks for JSON only. Each group gets a label, the ids of its comments, and quotes that are copied exactly from one of those comments. It never asks how many.
Below are viewer reviews, each prefixed by its id.
Group complaints that mean the same thing. Return a JSON array:
[{"label": "one-line summary", "ids": ["id", ...],
"quotes": [{"id": "id", "text": "a contiguous span copied from that review"}]}]
A quote must be copied character for character; if none fits, leave it out.
Do not write any counts.
The prompt asks nicely. The next two steps don't take its word for it.
Step 3: check every quote in code
A quote passes only if it is a contiguous substring of the review it cites. Normalise just two things: drop whitespace, and fold full-width punctuation to ASCII (models sometimes swap , for ,). Nothing fuzzier than that.
Step 4: count in code
"How many dramas have viewers saying this?" Count the distinct dramas behind the group's ids, after dropping any id that doesn't exist. Models do occasionally invent one.
Here are steps 3 and 4 run on a real model-shaped answer. One quote is verbatim, one is a paraphrase, one has an extra full stop, and one id is invented:
import unicodedata
reviews = [
{"id": "d1#0", "drama": "d1",
"text": "剧情不错,人物造型也养眼,就是对话部份错误较多,还有女主多处管谁都叫爸……"},
{"id": "d1#1", "drama": "d1", "text": "配角行为逻辑不通,看得很出戏"},
{"id": "d2#0", "drama": "d2", "text": "还可以吧,废话再少点就好了……"},
{"id": "d3#0", "drama": "d3", "text": "前面挺好,后面人设崩了,配角行为逻辑不通"},
]
by_id = {r["id"]: r for r in reviews}
# what a model returned (one group shown); note the rewrite and the made-up id
groups = [{
"label": "Plot holes and broken characters",
"ids": ["d1#0", "d1#1", "d3#0", "d9#4"],
"quotes": [
{"id": "d1#0", "text": "对话部份错误较多"}, # verbatim
{"id": "d1#0", "text": "对话错误很多"}, # paraphrased
{"id": "d3#0", "text": "配角行为逻辑不通。"}, # extra full stop
{"id": "d1#1", "text": "配角行为逻辑不通"}, # verbatim
],
}]
def norm(s: str) -> str:
s = unicodedata.normalize("NFKC", s) # full-width punctuation -> ASCII
return "".join(s.split()) # drop all whitespace
def verified(q: dict) -> bool:
src = by_id.get(q["id"])
return bool(src) and norm(q["text"]) in norm(src["text"])
for g in groups:
kept = [q for q in g["quotes"] if verified(q)]
ids = [i for i in g["ids"] if i in by_id] # drop invented ids
g["sources"] = len({by_id[i]["drama"] for i in ids})
print(f'{g["label"]}: {g["sources"]} dramas, '
f'{len(kept)} of {len(g["quotes"])} quotes kept')
for q in kept:
print(" ", q["id"], q["text"])
Output:
Plot holes and broken characters: 2 dramas, 2 of 4 quotes kept
d1#0 对话部份错误较多
d1#1 配角行为逻辑不通
Three things happened there:
-
The paraphrase was dropped. 「对话错误很多」 is not in review
d1#0. - The extra full stop was dropped too. The rule is strict on purpose: one extra character and it is no longer what someone wrote. Losing a quote is better than printing a lookalike in quote marks.
-
The invented id
d9#4was ignored before counting, so the "2 dramas" (d1 and d3) is a number code produced, not one the model wrote.
What it looked like on the real data
On the top 12 dramas of the AI-drama chart, the first group came out as:
Plot holes and characters falling apart (5 dramas): 「对话部份错误较多」「配角行为逻辑不通」
The 5 is a count of distinct dramas, done in code. Both quotes can be found word for word in the reviews they cite.
Summary
- The model groups and names. That is the part it is good at.
- Code counts (distinct sources, invented ids removed) and checks quotes (contiguous substring, only whitespace and full-width normalisation).
- A quote that fails the check is dropped. Don't ask the model to "write one that's closer".
This works for product reviews, interview notes and open-ended survey answers as well. Wherever a report shows quotes or "N of M", these two checks are worth the twenty lines.
Top comments (0)