What we learned shipping VS Code weekly (without breaking everything) | BRK204
Pierce Boggan and Joshua Spicer walk through how the VS Code team shifted from monthly to weekly releases without letting quality collapse, focusing on the concrete engineering patterns and tooling they adopted.
Overview
The session covers the VS Code team’s release-engineering and quality practices for shipping weekly at high commit volume, including how they use agents to turn conversations into pull requests, automate triage in a large GitHub repository, and apply evaluation harnesses and staged rollouts to reduce risk.
Topics covered
Moving from monthly to weekly releases
- Why the team moved to weekly releases
- What broke (or started to break) as release frequency increased:
- Test gaps becoming more visible
- Review bottlenecks
- Triage backlog scaling with throughput
Agent-driven workflows that fit into the team’s cadence
- Agent sessions before meetings to reduce coordination overhead
- Using conversations that become pull requests (PRs) rather than long specs
Agent verification loop for UI changes
- Demonstration: updating a UI button color (blue to green)
- Screenshot-based validation to confirm the change matches expectations
- A verification loop pattern to ensure UI changes are correct before merging
Automated triage and GitHub-scale repo hygiene
- Automated triage patterns used across one of GitHub’s largest repositories
- Duplicate issue detection
- Semantic matching to route and cluster issues more effectively
- GitHub integration improvements to reduce manual work
Telemetry-driven issue handling
- Automatic handling of unhandled errors using telemetry signals
- Connecting telemetry to GitHub workflows to create/route actionable items
Cross-process bug analysis and AI-generated fixes
- Cross-process VS Code bug analysis
- Using AI to propose fixes as part of the debugging workflow
Quality harnesses and offline evaluation
- Introduction of VSC-Bench as VS Code’s custom offline evaluation stack
- Harnesses intended to keep quality high while shipping frequently
Safer weekly releases with staged rollouts
- Staged rollouts as a risk-reduction mechanism for weekly updates
- How staged rollout practices complement high-frequency shipping