Content by Waldek Mastykarz (16)

Waldek Mastykarz explains how to evaluate agent skills that call APIs without paying for test traffic or mutating production data. He outlines why direct evals against real endpoints create cost, state, and non-determinism problems, then shows a proxy-based approach using Dev Proxy to emulate APIs while keeping production URLs unchanged.

Waldek Mastykarz explains how to design AI coding-agent evaluations (AX evals) that produce real signal instead of misleading scores, focusing on representative prompts, precise pass/fail/skip criteria, judge calibration, and enough repeated runs to separate variance from actual improvements.

Waldek Mastykarz explains why AI coding-agent eval scores often fail to reproduce across machines, and how “hidden variables” like OS, shell choice, workspace paths, and language-server feedback can quietly change an agent’s behavior and your measured results.

Waldek Mastykarz tests the common advice to replace complex CLI flags with a single --json payload for AI agents, and shows that traditional args can be more reliable and far cheaper in tokens—especially once retries and shell-escaping issues (PowerShell vs Bash) are factored in.

Waldek Mastykarz shares results from 150 GitHub Copilot Chat agent runs comparing Claude Sonnet 4.6 vs 5, showing how a “cheaper” model can still cost more due to token spikes and variance, and how output quality can regress depending on the task and available documentation.

Waldek Mastykarz explains why public coding benchmarks like SWE-bench don’t predict how well an AI coding agent will perform in your real developer environment, and outlines a practical approach for running representative, repeatable evaluations against your own scenarios, extensions, and harness.

Waldek Mastykarz explains what goes wrong when an AI coding agent has zero training data about your internal SDKs or proprietary frameworks, and how to design extensions that teach the model effectively. He focuses on baseline evaluation, avoiding “closest match” hallucinations, and balancing correctness against token and context-window costs.

Waldek Mastykarz explains why “model preferences” (like “Claude prefers React”) are usually artifacts of missing or skewed context, not stable traits of an LLM. He breaks down how prompts, formatting, and attached project files change outputs, and how to evaluate AI developer tools using realistic, repo-shaped tests.

Waldek Mastykarz explains why “skills” for AI coding agents often fail when they return large chunks of documentation the model already knows, wasting tokens and pushing out more valuable context. He outlines a measurement-first approach to define a lean skill scope that targets only the gaps where the model actually fails.

Waldek Mastykarz explains why AI coding agent extensions that perform well alone can degrade results when installed alongside other extensions, and how to measure and reduce these composition effects in real developer workspaces.

Waldek Mastykarz explains why AI coding agents often keep using deprecated tooling (like legacy CLIs) even after teams ship replacements, and outlines practical ways to measure and correct agent behavior using extensions, explicit deprecation signals, and better naming.

Waldek Mastykarz explains how AI coding agents can silently scaffold outdated Node.js projects when they run npx without pinning versions, due to npm’s engine-aware manifest selection. The post breaks down why this happens and gives practical steps to make agent-driven scaffolding more predictable.

Waldek Mastykarz explains how to measure whether an AI coding agent extension actually improves generated code, using controlled comparisons, clear evaluation criteria, and repeatable scenario runs while tracking both quality and token cost.

Waldek Mastykarz explains why piling up dozens of agent “skills” can quietly burn your token budget and reduce response quality, and how to decide what should stay a skill versus what should become a manually-invoked prompt in tools like GitHub Copilot and VS Code.

Waldek Mastykarz breaks down what happens between a developer prompt and an AI agent generating code with your SDK, CLI, or API, focusing on how harnesses assemble context, how models choose (or skip) tools like MCP servers, and why response quality and error messages can make agents succeed or fail.

Waldek Mastykarz breaks down the “agent stack” behind AI coding agents and explains why most problems aren’t in the model or the agent harness, but in the extension layer you can actually control. He outlines common failure modes and a practical way to measure whether your extensions create lift or drag.