Weekly ML Roundup: Governed AI, OneLake costs, real-time CDC

by TechHub

This week focused on making ML systems easier to run in production, from securing outbound AI calls in SQL Server 2025 to tightening governance and cost visibility across OneLake. Fabric updates added storage tiering and reporting, plus clearer architecture patterns for building an AI-ready data foundation. On the real-time side, Eventstreams matured with stronger connectors, event-time processing, and practical CDC-to-action reference flows, while new guidance covered context-aware AI layers and safer automation patterns for calling Fabric REST APIs.

This Week's Overview

SQL Server 2025: Securing outbound AI calls from the database

SQL Server 2025's outbound AI connectivity features are now generally available, and Microsoft focused this week on what it takes to make database-initiated AI calls safe to run in production, extending last week's MCP-based SQL Server agent tooling theme into SQL-native outbound controls and auditing. The GA set covers two key primitives: external REST invocation (via sys.sp_invoke_external_rest_endpoint) and external model objects (via CREATE EXTERNAL MODEL), which let SQL workloads call out to hosted models and APIs without pushing that logic into an app tier.

The practical story here is governance: you can keep outbound connectivity tightly permissioned, authenticated, and auditable, rather than having ad-hoc endpoints embedded in stored procedures or client code. The guidance emphasizes least privilege, restricting which endpoints can be called, and capturing activity using SQL-native controls like SQL Server Audit and Extended Events so security teams can monitor AI-related data egress the same way they monitor other sensitive operations.

OneLake and Fabric governance: cost controls, usage visibility, and architecture patterns

This week's Fabric updates clustered around making OneLake easier to run as an enterprise data foundation for AI workloads, building on last week's push for more governable, production-ready Fabric deployments by adding cost controls and clearer usage visibility alongside architecture guidance. The theme across the announcements is tightening the operational loop: understand where storage is going, put policies in place to control cost growth, and apply repeatable architecture patterns so downstream AI (including RAG pipelines) has consistent, governed data to work with.

OneLake Storage Tiers and Lifecycle Management (GA)

OneLake Storage Tiers and Lifecycle Management are generally available, adding hot/cool/cold tiers plus lifecycle rules that can automatically move data between tiers to reduce long-term storage cost while keeping it online in OneLake. The GA details include rule support like AutoTierFromCoolToHot, which is designed for cases where “cold” data unexpectedly becomes active again and needs to move back to a performance tier without manual intervention.

For teams building AI and analytics systems on Fabric, this helps prevent model training sets, feature tables, and log data from quietly expanding into expensive storage without a clear policy. It also encourages an operational posture where tiering rules become part of data product design, instead of a later cleanup step after bills spike.

OneLake storage report (GA)

The OneLake storage report is now generally available, giving workspace admins an item-size breakdown directly in the Fabric portal, which pairs naturally with last week's Item Recovery GA story since soft-deleted and retained artifacts are often where “mystery storage” shows up. The report includes visibility into visible vs. system data, soft-deleted data, and billing status, which matters when “mystery” usage comes from retained system artifacts or deleted-but-not-reclaimed items.

This is the kind of tooling that turns cost conversations into actionable work: you can identify the specific lakehouse items, tables, or artifacts driving spend and then decide whether to adjust retention, move tiers, or redesign ingestion. For ML teams, it is especially useful when experimentation creates many intermediate datasets that outlive their original purpose.

OneLake architectural guidance for AI readiness

Microsoft published a OneLake Architectural Guidance whitepaper that frames five repeatable patterns for modernizing data estates with AI in mind, continuing last week's OneLake shortcuts governance thread by showing how access, sharing, and domain boundaries fit into larger “AI-ready” designs. The guidance covers approaches like unifying data access, applying medallion architecture, building a governed data mesh, simplifying platforms, and enabling external data sharing without losing control.

The value for developers is having reference architectures that translate “AI-ready data foundation” into specific design patterns you can apply when setting up domains, access boundaries, and pipelines in Fabric. If you are standardizing how teams publish datasets for RAG, feature generation, or agent workflows, these patterns help reduce one-off designs that become hard to govern later.

Event-driven AI in Fabric: stronger connectors, event-time processing, and real-time “row-to-action” patterns

Fabric's Eventstreams updates this week were about moving from “we can stream data” to “we can operationalize it”, following last week's Eventstream connector and CDC preview momentum with more production-oriented operator support and end-to-end scenarios. The common thread is shortening the path from operational data change (CDC) to action, whether that action is an alert, a routed business event, or an updated real-time dashboard.

Eventstreams 2026 Q2 updates: connectors, SQL operator, and AI Skills

The 2026 Q2 Eventstreams update adds several GA capabilities, notably Kafka and Service Bus connectors and a GA SQL operator that supports event-time processing and testing. That combination makes it easier to build streaming topologies that behave correctly under out-of-order data (a typical real-world issue) while keeping validation closer to the canvas experience rather than relying only on downstream debugging.

Microsoft also introduced AI Skills aimed at generating and deploying eventstream topologies through Fabric REST APIs, including Git-based version control. For teams treating streaming logic as code, the story is maturing from click-built pipelines to repeatable, reviewable deployment workflows that still integrate with the Eventstreams authoring model.

Oracle database CDC source connector for Eventstream (Preview)

Microsoft added a preview Oracle Database CDC source connector for Fabric Eventstream, using Oracle LogMiner to capture row-level inserts, updates, and deletes, which is a direct follow-through on last week's Oracle CDC ingestion preview direction by making Oracle a first-class change source. Those changes can be routed in real time to Fabric targets such as Eventhouse (KQL), Lakehouse, and Activator, which is a practical bridge for enterprises where Oracle remains a core operational system.

For ML and real-time analytics teams, the key implication is lower friction for bringing operational change data into the same event-driven patterns already common for SQL and Kafka sources. It also opens the door to “freshness-sensitive” features (fraud signals, inventory changes, account events) being updated from Oracle without waiting for batch ETL windows.

Row-to-action scenarios: mirroring, Activator triggers, and AI enrichment in streaming

Two scenario posts drilled into the mechanics of turning database changes into immediate outcomes inside Fabric, expanding on last week's focus on production-ready ingestion by showing complete CDC-to-action reference flows. One walkthrough mirrors operational databases into OneLake using Delta Change Data Feed (CDF), streams changes through Eventstreams, then routes outputs to Activator for triggers and to Eventhouse for live KQL dashboards.

A second pattern extends this into “row-to-intelligent-action” by capturing Azure SQL CDC into Eventstreams, reshaping with DeltaFlow, enriching micro-batches in Spark Structured Streaming using AI Functions for PySpark, and publishing Business Events for routing and dashboards. Together, they outline a concrete architecture for combining CDC, streaming transforms, and AI inference without building a separate event platform from scratch.

Building context-aware AI on Microsoft platforms: IQ layers, orchestration, and secure API access

Several pieces this week focused on the less flashy but critical work of making AI systems context-aware and operable in enterprise environments, aligning with last week's “skills, tools, and permissions as the control plane” takeaway by emphasizing governed context and controlled API surfaces. The throughline is adding the right context (data, ontology, citations, memory), then providing secure, governed ways to orchestrate and call platform APIs so automation does not become a security or reliability problem.

Microsoft IQ Platform: Work IQ, Fabric IQ, and Foundry IQ

Microsoft described its IQ Platform as a layered approach to adding organizational and business context to AI systems: Work IQ, Fabric IQ, and Foundry IQ. The write-up highlights Fabric IQ as the OneLake-based data foundation paired with an ontology model, and Foundry IQ as the managed services layer for RAG-style retrieval, ranking, citations, and memory.

For developers building assistants or internal copilots, the key idea is that better prompts alone do not solve enterprise context. A shared ontology and managed retrieval/citation services can reduce the amount of custom glue needed to make responses traceable and grounded in approved data, which is often the difference between a demo and a system people can rely on.

Fabric orchestration and monitoring updates in Data Factory

Microsoft Fabric Data Factory received updates aimed at improving pipeline authoring and increasing monitoring and observability depth, with AI-powered operational capabilities positioned to streamline orchestration and troubleshooting, which builds on last week's automation/lifecycle push (API-driven agents and CLI migrations) by tightening the day-2 operations loop. While the post is not a changelog of every knob, the focus is on reducing time-to-diagnosis when pipelines fail and making it easier to understand behavior across runs.

This matters for ML teams because orchestration is where reliability problems surface first: feature pipelines, retraining schedules, and backfills all depend on consistent execution and clear monitoring signals. Better built-in observability in the Fabric authoring environment reduces the need to bolt on separate monitoring just to answer basic operational questions.

Securely calling Fabric REST APIs with User Data Functions

A guide outlined a secure pattern for invoking Fabric REST APIs from User Data Functions, using a service principal plus Azure Key Vault to centralize authentication and keep secrets out of application code, and it follows naturally from last week's theme of making agent and automation workflows safe by default through explicit identity and permission boundaries. The example use case includes governed operations like triggering pipelines, which can be a common need when teams want data-driven automation without granting broad interactive permissions.

For developers, the takeaway is treating REST automation as an identity-and-secrets problem first, then an API problem second. Using a service principal and Key Vault helps keep least-privilege boundaries clear, supports rotation, and avoids the “token in code” anti-pattern that tends to spread quickly across scripts and notebooks.

Other Machine Learning News

Microsoft shipped more operational knobs around enterprise data integration and network boundaries in Fabric, continuing last week's broader “production readiness” arc (governance controls plus operability) with tighter identity options and more predictable connectivity scaling. Dataverse Fabric Link now supports choosing which Dataverse tables sync into Fabric, adds workspace identity and service principal authentication, and reduces sync latency to under 15 minutes in most cases, which helps keep downstream features and reports closer to real time.

On the networking side, Fabric's Virtual Network Data Gateway autoscaling entered preview, letting admins set min/max node counts while Fabric adjusts capacity based on workload signals sampled every five minutes, with monitoring through diagnostics logs. For teams relying on DirectQuery or other VNet-bound access patterns, autoscaling can reduce the manual work of sizing gateways for peak load while still keeping execution inside controlled network paths.

A demo video showed what “real-time + AI” can look like when you push beyond dashboards into interactive operational experiences, echoing last week's real-time ingestion and control-plane focus by putting Eventhouse-backed telemetry and an agent experience on top of governed streaming. A 3D wind farm digital twin built on Fabric, combining LiDAR-derived geometry, live telemetry stored in Fabric Eventhouse, and a typed/spoken natural-language agent for spotting underperforming turbines and estimating cost impact. While it is a showcase, it usefully connects several Fabric building blocks (Eventhouse, real-time ingestion, and agent experiences) into an end-to-end scenario you can map to other asset-heavy domains.