Vague Wording and Conflicting Instructions

Make every instruction checkable

Words like "some", "appropriate", "good", "etc." and "as needed" leave the model to guess, which shows up as inconsistent outputs. Conflicting instructions ("brief" and "comprehensive", "return JSON" and "plain text") force arbitrary choices. Replace vague words with measurable instructions (3 bullet points, under 80 words, cite one source), and resolve conflicts by priority ("If the user asks for plain text, use plain text; otherwise JSON"). A simple scanner finds many of these.

Finding vague words and conflicts, run

I ran this with Python 3 (scipy 1.18.1 where imported) on example or seeded simulated data, not results from a real product. The example prompt contains the vague words some, appropriate and etc., and two conflicting pairs: brief versus detailed, and JSON versus plain text.

import re
VAGUE = ["some", "good", "appropriate", "etc", "as needed", "nice", "properly", "reasonable", "maybe"]
CONFLICTS = [(r"\bbrief\b|\bshort\b", r"\bdetailed\b|\bcomprehensive\b"), (r"\bjson\b", r"\bplain text\b")]
prompt = ("Write a brief but detailed and comprehensive reply with some appropriate examples etc. "
          "Return JSON. Reply in plain text if the user prefers.")
found = [w for w in VAGUE if re.search(rf"\b{re.escape(w)}\b", prompt, re.I)]
clashes = [(a, b) for a, b in CONFLICTS if re.search(a, prompt, re.I) and re.search(b, prompt, re.I)]
print("vague words:", found)
print("possible conflicts:", [(re.search(a, prompt, re.I).group(), re.search(b, prompt, re.I).group()) for a, b in clashes])

Output:

vague words: ['some', 'appropriate', 'etc']
possible conflicts: [('brief', 'detailed'), ('JSON', 'plain text')]

Turn adjectives into numbers

Rewrite "short" as "at most 60 words" and "a few examples" as "exactly 2 examples"; outputs become easier to score too.

Quick check: How should a conflict like "brief but comprehensive" be fixed?

• Leave it for the model to decide
• Replace it with a measurable instruction or an explicit priority rule
• Repeat both words more often
• Remove all instructions