Weekly ML Roundup: Fabric Spark GA, RAG reranking, AKS inference

by TechHub

This week's ML roundup spans practical platform work and model-facing guidance, from Fabric Runtime 2.0 reaching GA for Spark workloads to new Lakehouse targets for dbt jobs and better near real-time capacity monitoring in Real-Time Hub. On the application side, the RAG guidance makes a clear point: vector search is only candidate generation, and rank fusion plus reranking determine what your system actually uses. We also look at production-focused inference improvements on AKS with NVIDIA Dynamo and Blob Storage integrations, and a research update on CARE-X that combines vision-language modeling with calibration, grounding, and tool-augmented measurement for radiology workflows.

This Week's Overview

Microsoft Fabric leans further into Spark performance and Lakehouse-first workflows

Fabric Runtime 2.0 reaches GA for Spark-based engineering and data science

Fabric Runtime 2.0 is now generally available as the Apache Spark-based execution foundation behind Microsoft Fabric's Data Engineering and Data Science experiences, following last week's July 2026 feature rollup callout of Runtime 2.0 as a cue to start validating Spark version and behavior changes. The focus is on better performance plus tighter ecosystem integration, which matters if you depend on predictable Spark behavior for notebooks, jobs, and production pipelines.

For teams standardizing on Fabric, this GA milestone is a signal to validate runtime-dependent pieces like library compatibility, cluster sizing assumptions, and job performance baselines. It is also a good moment to revisit how you package dependencies and test transformations, since runtime changes can expose version-sensitive edges in Spark workloads.

dbt jobs add Fabric Lakehouse as a target (Preview)

Microsoft Fabric's dbt job now supports Fabric Lakehouse as a target in preview, extending last week's theme of OneLake-first foundations (including open table format support) by making Lakehouse a more direct execution surface for governed transformations. The announcement calls out Lakehouse-first scenarios and walks through configuration steps to connect to (or create) a Lakehouse target.

If your Fabric architecture is Lakehouse-centric, this reduces the friction of keeping a separate execution target just for dbt while still benefiting from dbt's testing and modeling conventions. Practically, it means you can align transformation logic with OneLake storage patterns and treat the Lakehouse as the primary interface for curated tables and downstream consumption.

Real-Time Hub Capacity Overview Events go GA for near real-time monitoring

Capacity Overview Events in Fabric Real-Time Hub are now generally available, enabling near real-time visibility into capacity conditions along with alerting and automated actions, and it builds on last week's Real-Time Hub and event-driven patterns by turning capacity state into another stream you can wire into ops workflows. This gives operations teams a more event-driven way to detect saturation and react before users feel it.

For developers running mixed workloads (ETL, notebooks, warehouses, streaming), near real-time capacity signals help you decide when to reschedule heavy jobs, apply backpressure, or triage runaway queries. Treat it as a building block for capacity-aware automation, especially if you already rely on event streaming patterns for ops telemetry.

SQL Audit Logs add identity-based predicate filtering to reduce noise

Microsoft Fabric SQL Audit Logs now support identity-based predicate filtering in GA, continuing last week's governance and security push (OneLake security GA and outbound controls) by making audit trails easier to filter into something reviewable at enterprise scale. Instead of wading through everything, you can filter for the identities and activity patterns that actually matter for investigations and compliance reporting.

This is most useful when you have many service principals, automation accounts, and user populations sharing the same Fabric estate. Less noise improves time-to-answer during incident response, and it can reduce downstream log storage and query costs if you export audit data into a central SIEM.

New previews push Fabric toward agent-assisted, end-to-end build workflows

Two Fabric previews target a common problem: AI helpers tend to stall when they lack domain-specific “how to” steps and safe ways to validate changes, and they pick up where last week left off on Fabric Data Agents and MCP-driven integrations by shifting attention from connectivity and lifecycle to day-to-day agent authoring. SQL database skills for Skills for Fabric add workload-specific guidance so AI coding agents can create, connect, query, and manage SQL databases as part of full Fabric solutions.

In parallel, “Build agent with AI” for Fabric Data Agent helps creators explore schemas, learn from query history, generate instructions and example queries, and validate changes using read-only queries before applying them. Together, these features are pointing toward safer agent authoring loops where the platform provides guardrails (instructions, examples, and validation) rather than expecting the model to infer everything.

RAG retrieval quality: vector search is only the first step

Vector search will get you a candidate set, but this week's RAG guidance puts the spotlight on what happens next: rank fusion and learned reranking decide what your app actually reads, and it complements last week's “governed context” platform direction by emphasizing that RAG quality comes from disciplined retrieval stages, not just picking a vector store. The walkthrough breaks down retrieval, Reciprocal Rank Fusion (RRF), and reranking, then maps those stages to real platform options across Azure AI Search, Azure SQL, PostgreSQL Flexible Server, and Azure Cosmos DB.

The practical takeaway is architectural: treat reranking as a first-class stage with its own latency and cost envelope, not a bolt-on after you pick a vector database. The demo uses a SQuAD-based setup and notebooks, and it compares common reranking choices like Azure AI Search's semantic ranker, Cohere Rerank v4.0 Fast, and Cosmos DB Semantic Reranker so you can reason about quality vs throughput trade-offs.

Faster LLM inference on AKS with NVIDIA Dynamo + Blob Storage integrations

An integration write-up details native connections between Azure Blob Storage and the NVIDIA Dynamo stack aimed at reducing LLM startup and serving latency on Azure Kubernetes Service (AKS), which fits the same “production-friendly systems” theme from last week by focusing on the operational bottlenecks (warmup and cache behavior) that show up after demos. Two specific performance targets are called out: faster model weight loading via Dynamo ModelExpress, and KV cache offloading via NIXL to reduce time-to-first-token (TTFT) in vLLM-style serving.

For teams running containerized inference, this is another step toward separating storage and serving concerns so you can keep weights in Blob while still avoiding slow pod warmups. It is especially relevant when you frequently scale deployments up and down or rotate models, because weight loading and cache behavior are often the hidden costs that dominate user-perceived latency.

Research: CARE-X pushes radiology VLMs toward clinical usefulness

Microsoft Research introduced CARE-X, a chest X-ray vision-language model designed to pair report generation with calibrated structured predictions and visual grounding, continuing last week's Research thread on making agent and model evaluation more reliable by adding more explicit calibration, grounding, and verifiable steps to the output. The training approach combines supervised fine-tuning with DAPO reinforcement learning, and the post frames this as a way to produce outputs that are easier to trust and verify in clinical settings.

A separate tool-augmented measurement pipeline uses Qwen3-VL-4B-Instruct to improve measurement-dependent diagnoses, acknowledging that some clinical judgments depend on precise measurements rather than purely generative descriptions. For developers building medical imaging systems, the interesting pattern is the hybrid design: use a VLM for perception and grounding, but route measurement-heavy steps through a tool-assisted pipeline that can be validated and audited.

Other Machine Learning News

Fabric operations and governance got a few pragmatic updates this month, building on last week's focus on operability (queue visibility, CMK for Spark, and OneLake security) by tightening the day-to-day mechanics of hybrid access and cost attribution. with the July 2026 on-premises data gateway release (v3000.326) aligning gateway behavior to the July 2026 Power BI Desktop query runtime and adding security, authentication, and diagnostics improvements. For hybrid estates, keeping the gateway and Desktop runtimes aligned is often the difference between predictable refresh behavior and hard-to-debug query mismatches.

Cost attribution is also getting clearer in Fabric warehouses: a new guide explains how to use Capacity Metrics and Query Insights together to attribute warehouse Capacity Unit (CU) consumption down to specific queries and users using vCore-seconds-based weighted attribution (including the queryinsights.exec_requests_history view). That gives teams a concrete way to tie performance and spend back to workloads, which is essential once multiple teams share a single capacity.

Azure platform updates touched AI-adjacent infrastructure too, including networking and edge changes (ExpressRoute, Azure Firewall Premium, Azure Front Door, and mTLS mentions), storage improvements like disk resizing for Premium SSDv2/Ultra, and Fabric recovery defaults. The same update also calls out AI model ecosystem movement, including Gemini 3.7 Flash appearing in GitHub Copilot coverage, which is a reminder to verify model availability and behavior if your dev workflow depends on specific Copilot-backed models.

For terminology clarity, a short refresher video explains how AI, machine learning, deep learning, and generative AI relate, and where large language models (LLMs) fit. It is a handy link to share internally when teams use the terms interchangeably but need a shared baseline.