Before You Ask AI to Recommend an Option, Write the Decision Rule
What a decision rule does
A decision rule turns a broad priority into visible instructions for evaluating evidence. It has four parts:
- **Criteria:** the dimensions used to compare the options.
- **Weights:** the relative importance of those criteria.
- **Disqualifiers:** conditions that remove an option from consideration even if it scores well elsewhere.
- **A tie-breaker:** the rule used when the evidence does not clearly separate the options.
For example, “prioritize predictable delivery” is a preference, not yet a usable decision rule. A team would still need to say what predictability means: perhaps implementation variance, dependency risk, support response, or another measurable factor appropriate to its situation.
The distinction is important. A criterion tells the researcher what evidence to collect. A weight tells the researcher how much that evidence should influence the recommendation. A disqualifier prevents a strong average from hiding an unacceptable condition. A tie-breaker makes ambiguity explicit instead of letting the model improvise.
Why the criteria must be visible
A recent research abstract illustrates the risk. The authors report 36 runs in which OpenAI- and Anthropic-based web research agents compared Microsoft and Alphabet. Among 24 reports conditioned on a stated priority, 21 focused on that priority, but the agents used different criteria and produced different rankings. Under a growth priority, they differed in whether they emphasized pace, risk, or predictability (arXiv:2611.22989).
That is a bounded finding, not a model leaderboard. The evidence comes from a small study described at the abstract level here. It does not show that every agent will behave this way, nor does it establish that one model makes better decisions.
It does support a useful process lesson: stating a priority does not guarantee that two researchers—or two AI agents—will operationalize it the same way.
When the criteria remain implicit, three things become hard to audit:
- whether the research gathered the right evidence;
- whether the final ranking follows from that evidence; and
- whether a different but reasonable interpretation of the priority would change the result.
Visible criteria do not eliminate judgment. They make judgment inspectable.
Build an AI decision brief in six steps
Use the following sequence before you prompt a research agent.
### 1. Name the decision and its owner
Write one sentence that identifies the choice and the person or group accountable for making it.
Why it matters: an AI system can organize evidence, but it should not silently become the decision owner. Naming the owner also clarifies who can resolve tradeoffs the evidence cannot settle.
### 2. List the options and the time horizon
State which options are in scope and when the decision must hold. If “do nothing,” “delay,” or “run a pilot” is a real option, include it.
Why it matters: a recommendation for the next quarter may differ from one intended to guide a three-year commitment. Hidden scope changes create false precision.
### 3. Translate the priority into criteria
Ask: “What would we need to observe to say this option serves the priority?” Turn the answer into criteria that can be supported by evidence.
If the priority is predictable delivery, useful criteria might include dependency uncertainty, implementation variance, or the availability of support evidence. Those are illustrative categories, not universal measures. Your team must define them for its own decision.
Why it matters: the research agent now knows what evidence is relevant and what would merely be interesting background.
### 4. Add weights, disqualifiers, and a tie-breaker
Record which criteria matter most. Then state any unacceptable condition and what should happen when evidence is incomplete or options are close.
A tie-breaker might be “prefer the option with stronger directly verifiable evidence.” A disqualifier might be “exclude any option that fails a required compliance review.” These are hypothetical examples; they are not recommendations for a particular procurement.
Why it matters: without these instructions, the agent may invent its own way of resolving conflict between speed, cost, risk, and evidence quality.
### 5. Require an evidence log
For every material claim, ask for the source, publication date, source type, and a note explaining how the evidence affects a criterion. Separate verified facts from the agent’s interpretation.
Why it matters: a polished synthesis can hide a weak chain of evidence. A source log lets a human reviewer inspect both the claim and the role it plays in the recommendation.
### 6. Ask for sensitivity, not just a winner
Have the agent show whether the recommendation changes when a major weight changes or uncertain evidence is removed.
Why it matters: if a small change flips the ranking, the important output is not “Option A wins.” It is “this decision depends on an assumption the owner must resolve.”
A bounded practice exercise
Here is a hypothetical way to test the method without turning the exercise into a benchmark.
Choose one reversible, low-stakes decision with two options. Write the decision brief first. Then run the same research request twice: once with only the broad priority and once with the explicit criteria, weights, disqualifiers, tie-breaker, and evidence-log requirement.
Compare the outputs on process questions:
- Did both versions use the same meaning of the priority?
- Can a reviewer trace each material claim to a source?
- Does the recommendation follow the stated decision rule?
- Which missing evidence could change the choice?
Do not use one exercise to claim that a model is generally reliable or superior. The aim is to expose ambiguity in your workflow, not to manufacture a performance score.
Common mistakes
Treating the ranking as objective truth. A ranking reflects the criteria, weights, evidence, and unresolved uncertainty supplied to the system. Change those inputs and the ranking may change.
Writing criteria that already contain the conclusion. “Ability to prove Vendor A is safer” is not a neutral criterion. Define the condition to evaluate, then let the evidence speak.
Hiding source quality inside a summary. Vendor documentation, independent research, and an unverified claim do not carry the same evidentiary weight. Label the source type and limitations.
Using numbers to disguise judgment. A weighted score can make a decision look precise even when the underlying evidence is weak. Keep uncertainty visible and do not assign confidence the source cannot support.
Skipping the human decision owner. The agent can identify tradeoffs. It cannot accept organizational accountability for them.
The takeaway
Before asking an AI researcher which option is best, define what “best” means for this decision.
Name the criteria, weights, disqualifiers, tie-breaker, evidence requirements, and human owner. Then ask the system to show not only its recommendation, but also which assumptions could change it.
The goal is not to make AI recommendations sound more certain. It is to make the reasoning visible enough for a person to challenge, revise, and own.
Source and evidence boundary
- Primary approved source: arXiv:2611.22989.
- Verified at the approved-research stage: the abstract reports 36 runs; 24 were priority-conditioned; 21 focused on the stated priority; criteria and rankings varied.
- Interpretation: the tutorial’s decision-brief method is a process recommendation derived from that bounded signal.
- Not established: broad model superiority, universal agent behavior, procurement outcomes, return on investment, or causal performance improvement.
Member discussion