Your AI Agent Finished the Deck. Now Prove the Artifact Works.

A finished deck is not a finished task.

The file may open. The charts may look polished. The narrative may sound confident. None of that tells you whether the artifact is safe to put in front of a steering committee.

That distinction matters as AI agents move beyond chat and start producing the files teams actually use: plans, status packs, spreadsheets, presentations, and reports. Google DeepMind and Artificial Analysis recently introduced the Multimodal Knowledge Agent benchmark, or MKA, to test agents on visually complex knowledge-work files. The launch materials describe 200 private tasks across 10 verticals and 83 occupations, with expert rubrics weighted 75% toward factual accuracy and 25% toward visual design. Ten tasks are public. The same harness, tools, sandbox, and turn budget are used across tested models. Google DeepMind launch · MKA technical paper · BenchLM listing

MKA is useful evidence about one benchmark design. It is not proof that an agent will improve your team’s productivity, and its private task set cannot be independently inspected in full. The practical lesson is narrower and more valuable: evaluate the artifact, not the fluency of the demo.

Here is a seven-part acceptance test for the artifacts PMs and TPMs actually ship.

1. Did the artifact complete the intended task?

Start with the outcome, not the file type. “Create a roadmap deck” is too vague. A better task definition says who will use the deck, which decision it must support, which inputs it must reconcile, and which sections are mandatory.

Write the acceptance criteria before the run. For a weekly status pack, they might include every active workstream, current milestone dates, named owners, material risks, and a clear decision request. If the artifact omits a required section, it did not pass—even if the slides are beautiful.

2. Are the facts traceable and correct?

Check dates, totals, owners, labels, quotes, and status statements against the supplied sources. Separate an unsupported claim from a formatting error: both matter, but they require different fixes.

For repeatable evaluation, create a small answer key. List the critical facts that must appear and the source for each one. Then record false statements, missing facts, and statements that cannot be traced. A plausible sentence without a source is not a passing fact.

3. Is the visual representation accurate?

Visual quality is not decoration. A chart can use the right numbers and still communicate the wrong conclusion through a truncated axis, mismatched legend, misplaced label, or inconsistent time scale.

Inspect charts, tables, hierarchy, overflow, contrast, and page-level consistency. Ask whether a reader can distinguish actuals from forecasts, risks from issues, and committed dates from targets. Review the rendered output, not only the underlying text or formula cells.

4. How many turns did success require?

The MKA launch reports average turn counts ranging from 30 to 130 in its tested setup. That number is specific to the benchmark, but the measure is broadly useful. Google DeepMind launch · MKA technical paper · BenchLM listing

Record the number of agent turns, tool calls, and human corrections required for an accepted artifact. A workflow that succeeds after repeated rescue prompts may still be useful, but it is not equivalent to a reliable one-shot process. Turns also expose ambiguous instructions: if every run needs the same correction, the task definition or template probably needs repair.

5. What did the accepted result actually cost?

Do not compare a single model call with a completed human workflow. Capture the cost of the full accepted run: model usage, tool or rendering charges, retries, and human review time.

This is also why headline cost rankings should not travel unexamined from a benchmark into an internal business case. Costs depend on the harness, model version, prompts, access tier, and failure rate. Use the benchmark to choose what to measure; use your own representative tasks to make the decision.

6. What failed, and how many retries followed?

“Eventually passed” hides operational risk. Keep a retry log with a reason for each failure: incorrect fact, broken formula, missing section, rendering defect, tool error, or reviewer rejection.

Patterns matter more than isolated mistakes. If a model repeatedly loses citations when converting a report into slides, that is a workflow constraint. If a spreadsheet agent overwrites formulas during formatting, the task needs a protected range or a different approval boundary.

7. Where did a person have to intervene?

Record each intervention and why it was necessary. Useful categories include clarifying the task, supplying missing evidence, correcting a claim, repairing a visual, resolving a tool failure, and making the final judgment.

Human intervention is not automatically a failure. It becomes a problem when it is invisible, unpredictable, or assigned to nobody. The evaluation should tell you which decisions remain human and whether the workflow makes those decisions easy to inspect.

Build a small test before a broad pilot

You do not need 200 tasks to learn something useful. Start with five to ten recurring artifacts that represent real work. Preserve their source files, expected outputs, and reviewer rubrics. Include at least one messy case: conflicting dates, a missing input, a chart that needs interpretation, or an ambiguous status.

For each artifact, define:

1. Required sections and decision outcome.
2. Critical facts and their sources.
3. Visual checks.
4. Maximum acceptable retries.
5. Cost and elapsed-time fields.
6. Human intervention categories.
7. A named reviewer and pass threshold.

Run the same test set when the model, prompt, toolchain, or template changes. Record the version and date. Otherwise, a better-looking second run can quietly become a different experiment.

Decide with an acceptance table, not a demo reel

A simple decision table keeps the conclusion honest. Mark each task pass, conditional pass, or fail. Add the reason, retry count, total cost, and required intervention. Then decide where the agent belongs:

• Assist: a person drives; the agent drafts or transforms.
• Execute with approval: the agent produces the artifact; a named reviewer must approve it.
• Bounded automation: the agent may complete a narrow recurring task within explicit limits and monitoring.
• Not ready: failures are too frequent, too hard to detect, or too costly to recover.

The point is not to crown a universally best model. It is to learn whether a specific system can produce a specific artifact under conditions your team can govern.

Choose one recurring artifact this week. Write its acceptance rubric before the next agent trial. If you cannot say what “passed” means, the agent is not the only part of the workflow that needs evaluation.

Sources and limitations

Primary and technical references: Google DeepMind launch · MKA technical paper · BenchLM listing. The 200-task evaluation is private; only 10 tasks are public. Sponsor-reported results, autorater agreement, rankings, and cost claims are not treated here as independent evidence of general PM productivity or business outcomes.