When prompting is not enough for your agent
Microsoft Developer focuses on improving an AI agent through a measured optimization loop rather than repeated manual prompt edits.
Overview
The video demonstrates how to use Agent Optimizer in Microsoft (Azure) AI Foundry to iteratively improve an agent’s instruction/configuration with evidence-based evaluation.
What problem this addresses
- The agent behaves inconsistently (misses details, confirms orders too early, etc.).
- Rewriting instructions repeatedly becomes guesswork.
- The goal is to improve the agent’s configuration (instructions) without retraining the underlying model.
Agent optimization workflow shown
1) Start with evaluation data and success criteria
- Create a representative evaluation dataset.
- Define clear success criteria so changes can be measured.
2) Establish a baseline
- Score the current agent against the dataset.
- Use this as the baseline for comparison.
3) Create a new optimization
- Create a new optimization run in Foundry.
- Configure the optimization approach.
4) Choose models for optimization and evaluation
- Select candidate models (used to generate candidate instruction variants).
- Select evaluator models (used to score candidates against the criteria).
5) Use production traces as input data
- Use production traces as data to ground evaluation in real usage patterns.
6) Review estimated costs
- Review the estimated optimization costs before running.
7) Run the optimizer and compare results
- Run the Agent Optimizer.
- Compare baseline vs candidate instruction sets.
- Review what changed in the strongest candidate.
8) Validate quality, latency, and safety signals
- Analyze quality and latency impacts.
- Review evaluation scores.
- Inspect passed and failed tests.
- Look for sensitive information disclosures surfaced by evaluation.
9) Final check before shipping
- Validate the strongest candidate on held-out cases (not used during optimization).
- Ship a new agent version only after confirming improvements hold up.
Video timeline
- 0:00 — Fix inconsistent agent responses
- 0:36 — Introducing agent optimization
- 1:07 — Create a new optimization
- 1:41 — Configure the optimization approach
- 2:14 — Select candidate and evaluator models
- 2:47 — Use production traces as data
- 3:20 — Review estimated optimization costs
- 3:54 — Run the Agent Optimizer
- 4:29 — Compare baseline and candidates
- 5:02 — Analyze quality and latency
- 5:35 — Review evaluation scores
- 6:08 — Inspect passed and failed tests
- 6:40 — Find sensitive information disclosures
- 7:14 — Improve your agent from results
- 7:47 — Agent Optimizer recap and next steps