Weekly ML Roundup: Governed Context, Faster Spark, Real Agents

by TechHub

This week's ML roundup focuses on making AI systems more reliable by tightening the connection between governed data and agent grounding, from Fabric's lineage-first item relations API to table-level discovery in OneLake Catalog search. We also saw practical Spark work that pushes latency down and correctness checks earlier, including Structured Streaming real-time mode for fast quarantine and native JSON parsing in Fabric's Native Execution Engine. Finally, three production case studies (supply chain planning, closed-loop lab discovery, and feedback-to-work-item automation) show what agentic systems look like when they ship with traceability, safety controls, and human review.

This Week's Overview

Fabric's push toward governed context for AI (lineage, search, and meaning)

This week, Microsoft Fabric kept tightening the loop between data governance and AI agent grounding, with previews that expose richer context and make it easier to find the right assets in OneLake, continuing last week's thread on policy-driven OneLake governance and MCP-connected agent tooling by adding more machine-readable context an agent (and your CI/CD) can actually use. The common thread is moving from “an agent can read data” to “an agent can understand what data is, where it came from, and what will break if it changes”.

For teams building copilots and automated workflows on top of Fabric, these updates point toward a more reliable architecture: use catalog search to discover assets, use lineage graphs to constrain and explain what the agent touches, and use semantic/ontology layers to standardize meaning across domains. If you already run CI/CD for Fabric items, the new APIs and settings add concrete hooks to turn governance rules into automated checks.

Lineage-aware AI with the Fabric item relations API (Preview)

Microsoft Fabric introduced a preview item relations REST API that returns upstream and downstream dependencies as a typed graph, not just a flat list, building on last week's push to make governance operational (not just documented) by giving you a programmatic lineage surface you can wire into validation and agent grounding. That matters because you can programmatically answer questions like “what feeds this semantic model?” or "what reports will be impacted if this lakehouse table changes?", then use the result both for impact analysis and as grounding context for an agent.

In practice, this enables automated documentation and catalog enrichment, pre-deploy validation in CI/CD (for example, block a release if a dependency is missing or points to the wrong workspace), and safer agent tool use (constrain actions to the dependency neighborhood of an approved item). Because the output is structured, you can treat lineage as code: diff it across environments, gate changes, and create alerts when critical paths change.

Table discovery in OneLake Catalog search (Preview)

OneLake Catalog search (preview) now returns table-level results, both in the global search UI and via the OneLake Catalog Search API, which extends last week's OneLake catalog and mirroring story by making the “what tables exist where” question easier to answer before anyone tries to query or mirror more data. This is a practical improvement for large tenants where users (and agents) often know the table name they need but not the lakehouse/warehouse it lives in.

The preview adds a tenant setting controlling whether users can discover objects in search, and it clarifies how control-plane permissions and object-level security (OLS) affect what is visible. The richer filters and query operators are also notable for automation: you can build internal discovery tools, wire search into Fabric MCP servers, or implement “find the right table, then verify access” flows before an agent attempts a query.

Ontologies and the “meaning layer” for domain AI readiness

A separate thread this week argued that AI readiness often fails at the meaning layer, not the storage layer, especially in regulated, multi-vendor domains like oil and gas, echoing last week's emphasis on RAG as the grounding layer by arguing you cannot ground reliably if the domain concepts themselves are inconsistent. The post connects OSDU and Azure Data Manager for Energy (ADME) with Fabric IQ's preview ontology capability, positioning ontology-backed semantics as the bridge between cross-domain reasoning and governed agent grounding.

The developer takeaway is that semantic normalization needs to be first-class if you want reliable RAG and agent workflows: consistent business definitions, relationships, and constraints help prevent “correct query, wrong meaning” outcomes. Practically, that points toward combining OneLake storage with Power BI semantic models (for analytics semantics) and an ontology layer (for domain entities/relationships) so agents can use governed concepts instead of guessing from column names.

What the various “IQ” products are trying to solve

Marco Casalaina's walkthrough of Foundry IQ, Fabric IQ, Work IQ, and Web IQ framed them as grounding systems that connect agents to different context sources, following last week's Work IQ/Foundry IQ/Fabric IQ framing by clarifying the boundaries between context providers rather than treating grounding as a single feature. Instead of treating grounding as one generic feature, the “IQ” split makes it clearer which source of truth your agent is allowed to consult for a given task.

For developers, this helps with architecture decisions: Fabric IQ implies data and metadata context in OneLake and Fabric, while Work IQ implies permissions and context from Microsoft 365 apps. If you are mixing multiple sources, the important design work becomes deciding what gets retrieved from where, and how to keep the agent's citations and actions aligned to the governance model of each system.

Faster, safer data pipelines for ML workloads (Spark streaming and native execution)

Two Spark-related writeups landed on the same theme: push latency down and push correctness checks earlier, especially where ML feature pipelines and anomaly detection depend on fresh, trustworthy events, and they pair naturally with last week's Fabric Runtime 2.0 baseline by showing where teams are trying to spend that newer Spark headroom on real-time quality and ingestion performance. One focused on Structured Streaming latency and stateful processing patterns, while the other focused on raw performance improvements in semi-structured ingestion.

If you run feature computation, monitoring, or real-time scoring in Spark, the combination is straightforward: use lower-latency streaming modes to detect and quarantine bad events immediately, and use native JSON paths to reduce parse overhead and JVM transitions in lakehouse ingestion. Both approaches reduce the time between “event arrives” and “model or alert can safely use it”.

Sub-millisecond anomaly detection with Spark Structured Streaming Real-time mode

A Databricks experiment showed how Spark Structured Streaming Real-time mode can hit sub-millisecond anomaly detection latency by avoiding micro-batch waiting behavior, complementing last week's event-driven Fabric patterns by showing what the validation layer can look like once streams are truly low-latency. The example pipeline consumed Ethereum events and quarantined corrupted or sensitive records using protocol invariants plus payload scanning, demonstrating a pattern that applies to any high-volume event stream with strong validation rules.

For practitioners, the useful details are the stateful design choices: using APIs like transformWithState to track invariants and make per-event decisions without waiting for a batch boundary. This is especially relevant for ML ops pipelines where data quality issues can silently poison features; early quarantine keeps the “clean” stream flowing while preserving a forensic trail of rejected events.

Native JSON parsing in Fabric's Spark Native Execution Engine (Preview)

Microsoft Fabric's Spark Native Execution Engine (preview) added native JSON reading and parsing using a Velox/Gluten-based columnar execution path, extending last week's Fabric performance focus (including GPU query acceleration previews) into the ingestion side where semi-structured data often dominates. The key promise is fewer JVM transitions and better end-to-end performance for semi-structured lakehouse pipelines, where JSON ingestion and schema handling often become bottlenecks.

For ML teams, this matters when raw events land as JSON and need to be normalized into Delta tables for training or near-real-time features. If you are already using Fabric lakehouses, this preview signals that “JSON-heavy ingestion” is becoming a first-class performance target, which can simplify pipeline design (less pressure to pre-transform outside Spark) while keeping compute costs predictable.

Real-world agentic AI systems: supply chains, labs, and customer feedback

Three case studies this week showed what “agentic” looks like when it is tied to operational systems: decision support in supply chains, closed-loop experimentation in chemistry, and feedback-to-work-item automation inside Azure, reinforcing last week's shift from assisted authoring to tool-connected automation by showing the controls (traceability, safety, review) that make these systems shippable. The common lesson is that agents become more useful when they operate with guardrails: governed data sources, content safety, audit trails, and human review where needed.

For developers, the practical angle is architectural: these are not single-model demos, but workflows that combine data platforms (Fabric, Databricks, ADX), orchestration and compute (AKS), LLM endpoints (Azure OpenAI), and controls (Content Safety, human-in-the-loop, DevOps traceability). If you are building internal agents, these examples provide patterns for grounding, evaluation, and operational integration.

Kinaxis Maestro: scenario modeling and agentic experiences on Azure

Kinaxis described how its Maestro supply chain platform uses Azure building blocks like AKS, Azure Databricks, Azure OpenAI, and Cosmos DB, plus Azure AI Content Safety, aligning with last week's Databricks cost and operability guidance by highlighting the platform mix teams actually run when agents sit on top of data pipelines. The product focus is scenario modeling and forecasting under uncertainty, with agentic experiences layered on top to help users explore options and outcomes.

An implementation detail worth noting is the blend of data and application infrastructure: Databricks for data processing, Kubernetes for services, and LLMs for interaction and reasoning. Kinaxis also highlighted internal use of GitHub Copilot to support PR-based agentic development, reinforcing the “agents in the product” plus “agents in the engineering workflow” pattern.

Microsoft Discovery: closed-loop molecular design with durable memory

Microsoft Discovery was used for a closed-loop molecular discovery campaign for organic redox flow batteries, combining agentic AI, computational property predictors, and wet-lab feedback, and it extends last week's “grounding layer” discussion into science where grounding must include experimental provenance and negative results. A key workflow detail is preserving negative results in Discovery Bookshelf so future iterations do not repeat failed directions, which is often a missing piece in R&D pipelines.

For ML engineers, this maps to a familiar concept: iterative optimization with tight feedback loops, but with the added constraint that the “dataset” includes experimental outcomes and provenance. The mention of GraphRAG and durable knowledge stores is a reminder that in scientific domains, grounding must include not just papers and theories, but what actually failed in your lab.

Azure GEM: turning fragmented feedback into auditable work items

Azure's GEM (Great Experiences Matter) initiative described agentic workflows that aggregate customer feedback from multiple channels, produce prioritized insights, and generate auditable Azure DevOps work items, which builds on last week's emphasis on validation and logging for agent outputs by showing a concrete traceability model tied to delivery systems. The workflow keeps humans in the loop and can optionally use GitHub Copilot-assisted remediation for straightforward fixes, which helps connect “insight generation” to real engineering throughput.

The architecture calls out Microsoft Foundry, Azure Data Explorer, and Microsoft Fabric as data and analysis components, plus tooling integration via Azure DevOps. If you are building similar pipelines, the important pattern is traceability: every suggestion and action can be tied back to source feedback and tracked through a standard delivery system, which is essential for accountability and for measuring whether fixes actually improved outcomes.

Other Machine Learning News

Several platform updates this week were not strictly “ML features”, but they affect how teams ship ML-powered apps by improving the surrounding data and developer workflow, continuing last week's theme of turning governance and workflows into everyday developer paths (for example, CI/CD and operational controls) rather than side documentation. Microsoft Fabric advanced its SQL Query Editor with a GA results grid, a redesigned object explorer, and IntelliSense improvements for large warehouses, plus query sharing/management and pathways into OneLake analytics (Spark/KQL), Direct Lake semantic models, and Fabric Activator alerting (preview).

On the operational side, automatic index compaction entered public preview for Azure SQL Database, Azure SQL Managed Instance, and SQL database in Microsoft Fabric, aiming to reduce traditional index maintenance work. For agent developers in .NET, a tutorial showed how to build with the .NET Microsoft Agent Framework using a local Ollama model, then run local evaluation with LocalEvaluator checks like KeywordCheck and ToolCalledCheck to validate both response quality and tool use. Finally, an early look at Fabric Apps outlined how teams can replace shared Excel trackers with operational apps inside Fabric, which pairs naturally with Copilot-style workflows when the underlying data model is governed.