Weekly ML Roundup: Fabric Agents, Governance and Copilot Controls

by TechHub

Microsoft introduced Project Osmos for Fabric, which coordinates data engineering agents and runs persistent PySpark tasks after local assistants disconnect, with execution governed by user permissions and Fabric capacity consumption. OneLake catalog's Govern experience became generally available, centralizing estate-wide oversight, while preview policies add explicit controls over who can edit governed workspace settings. Paid GitHub Copilot users in VS Code can now choose low, medium, or high eagerness for Next Edit Suggestions, balancing smaller interruptions against more extensive proposed edits. Fabric Data Warehouse added preview Python function calls from T-SQL and workspace-level query monitoring, while local Qwen3.8-27B testing showed that higher request concurrency can reduce completed agent work. Together, these developments emphasize persistent agent execution backed by access controls, runtime monitoring, and workload-specific tuning.

This Week's Overview

Fabric dashboards and application visuals gain more presentation controls

Microsoft Fabric added generally available dashboard design tools and preview React components for analytics applications. The updates serve different workflows: arranging dashboards on a canvas and building visualizations into application code.

Dashboard layouts and multi-KPI visuals

Design Tiles, multi-KPI visuals, and new key performance indicator (KPI) styling capabilities are generally available in Fabric Real-Time Dashboards. Design Tiles provide Title, Text Box, and Divider elements with on-canvas editing, giving authors explicit controls for organizing and explaining dashboard content. Multi-KPI turns labeled query rows or numeric columns from a single row into multiple indicators within one visual.

Individual KPI tiles and multi-KPI visuals support Number, Bar, Gauge, and Donut styles. Shared configuration covers value formatting, scale, thresholds, colors, and directionality, so authors can apply consistent interpretation rules across indicators. Developers can choose the query shape that fits their data rather than creating a separate visual for every metric.

React components for agent-generated analytics

Fabric visuals and Fabric data grid are preview React packages available on npm for use in Fabric App templates or other React applications. Fabric visuals accepts Vega-Lite specifications that AI coding agents can generate from natural-language prompts, while both components use Fabric visuals core for shared input data, interaction messages, and theming. The grid supports filtering, sorting, column reordering and resizing, data bars, images, and extensible cell rendering, providing application controls beyond chart generation alone.

Fabric governance centralizes oversight and settings controls

Fabric's governance updates bring estate-wide oversight and workspace editing policies into OneLake catalog. Their availability differs: the Govern experience is generally available, while Policies Center and the workspace settings editing policy remain in preview.

OneLake catalog Govern and Govern Skills

The generally available Govern experience centralizes governance for Fabric tenants, domains, capacities, workspaces, tags, and policies. Govern Skills are now available in Skills for Fabric across the Health, Protect, and Trust governance pillars. They use supported Fabric admin APIs and request confirmation before sensitive or destructive actions.

The governance insights report provides a customizable semantic model and is available to every tenant regardless of license or size. Administrators can use that model to adapt reporting to their governance needs rather than relying only on fixed views. Policies Center remains in preview and supports attribute-based controls across tenants, capacities, domains, and workspaces.

Explicit permission to edit governed settings

The preview Allow workspace settings editing policy initially covers customer-managed key encryption and outbound networking controls. Administrators can define rules by Microsoft Entra security-group membership, workspace, and setting type, but an allow rule does not grant privileges: users must still be workspace admins. Fabric enforces the rules through both the portal and APIs, records decisions in audit logs, and centralizes the policies in OneLake catalog.

Advanced mode blocks governed edits that match no allow rule. Administrators should therefore review related settings together, because allowing encryption edits alone can block managed private endpoint edits.

Fabric data engineering adds persistent agents and alternative data paths

Fabric data engineering coverage spans agent-run Spark tasks, access to versioned datasets, and a Rust-native Spark-compatible execution option. Each has a different operational boundary to check: capacity consumption, read-only access, or workload compatibility.

Project Osmos coordinates persistent engineering tasks

Project Osmos coordinates specialized agents that investigate data, generate code, validate outputs, and refine implementations in Microsoft Fabric. It submits PySpark through the Fabric Livy API for execution in Fabric Spark sessions. Tasks persist after users disconnect their local coding assistants, so execution is not tied to an open local session.

Persistent tasks consume Fabric capacity units, making task lifetime an operational consideration as well as a convenience. Agents operate within user permissions, with Fabric and OneLake enforcing authorized access. Getting started requires Contributor or higher workspace permissions, the required tenant settings, and eligible Fabric capacity; trial capacities are unsupported.

Versioned lakeFS datasets through OneLake shortcuts

Fabric can read lakeFS-managed datasets through OneLake S3-compatible shortcuts. In the validated configuration, branch-following shortcuts reflected new commits, while commit-pinned shortcuts kept returning the original, byte-identical data. This gives developers a choice between following dataset updates and holding a stable input for reproducible work.

Delta table shortcuts created under Tables can be queried through the Fabric SQL analytics endpoint. The validated setup could not write changes back to lakeFS, and Microsoft's S3-compatible shortcut documentation specifies access-key and secret-key authentication.

Spark-compatible processing with Sail

Sail implements Spark Connect for existing Spark SQL and DataFrame workloads targeting Spark 3.5 or 4.x, subject to feature compatibility checks. Its OneLake catalog provider supports Iceberg REST and Unity Catalog-compatible Delta metadata endpoints. The tutorial authenticates with a Microsoft Entra service principal, writes revenue aggregates to an Iceberg table, and uses an Arrow Python user-defined function (UDF) to generate sentence embeddings from product reviews.

LakeSail Cloud Sessions proxy private compute through scoped, time-limited tokens. Developers should validate their workload's feature coverage before switching execution engines, and should not treat the reported nearly 98% cost reduction as a measured OneLake saving.

Copilot lets developers tune inline suggestion behavior

Paid GitHub Copilot subscribers using VS Code with Next Edit Suggestions enabled can now choose low, medium, or high eagerness. A single model supports all three modes, controlling how often suggestions appear and how much code they propose changing. On-policy distillation, a training technique, preserves medium-mode behavior, while reinforcement learning rewards the low and high modes for relative edit distance and suggestion quality.

Compared with medium, low eagerness produced a 29% lower dismissal rate and 28% shorter suggestions. High eagerness showed 40% more suggestions, produced 22% longer suggestions, and increased accepted and retained characters by 40%. These results describe a practical trade-off between fewer, smaller interruptions and more extensive proposed edits, rather than one setting that suits every workflow.

Fabric Data Warehouse expands execution and monitoring options

Fabric Data Warehouse previews address custom Python execution, distributed scalar function execution, and query monitoring. These are separate tools for extending warehouse logic and understanding its runtime behavior.

Calling Python from T-SQL

Data Warehouse now integrates with Fabric User Data Functions in preview, allowing T-SQL external functions to reference published Python functions. Queries can invoke them from SELECT, INSERT, and UPDATE statements, with execution delegated to the Fabric User Data Functions runtime. Supported scenarios include accessing OneLake data, retrieving Azure Key Vault secrets, and encrypting or masking sensitive columns.

The example uses Python's ipaddress library to calculate and persist IPv6 /48 network prefixes for downstream analytics. It illustrates how developers can bring Python logic into warehouse statements without expressing that logic entirely in T-SQL.

Broader scalar UDF inlining, with stricter creation checks

Expression Block inlining compiles and executes procedural scalar UDF logic across distributed analytical workloads. The update adds BREAK and CONTINUE loop control and supports these scalar function calls inside IF EXISTS queries, with existing functions gaining supported capabilities without alteration.

CREATE OR ALTER FUNCTION now rejects non-inlineable definitions by default and identifies the blockers in its errors. WITH INLINE = AUTO permits intentionally non-inlineable standalone functions, but does not make them inlineable or change their execution behavior.

Workspace-level warehouse query monitoring

The workspace monitoring preview covers warehouses, SQL analytics endpoints, and warehouse snapshots. Enabling it creates a read-only monitoring Eventhouse and KQL database scoped to the workspace, where WarehouseExecutions records completed, user-initiated distributed queries with duration, CPU time, status, result codes, and identity data.

WarehouseExecutions OperationId matches Query Insights distributed_statement_id, allowing exact lookups of corresponding query records. Administrators can save Kusto Query Language (KQL) queries or build Real-Time Dashboards to track performance percentiles, CPU usage, and recurring failures.

Eventstream separates deployment references from network access

Eventstream's previews address environment-specific configuration and private event publishing. Both require preparation in the target workspace, but they solve distinct deployment and security problems.

Reusable resource references across deployments

Variable library integration supports Eventstream source connections and destination items through connection-reference and item-reference variables. These point to managed Fabric resources rather than storing credentials or raw connection strings, and their symbolic references survive Git integration and deployment pipelines.

Only one variable library value set is active at a time, and Eventstream resolves references against that set. Before deploying and publishing, teams must ensure that the target workspace has accessible resources and correctly configured variable libraries.

Private publishing to individual workspaces

Workspace-level private links now support Eventstream custom endpoint sources in preview. Applications can publish through private connections using supported Kafka or AMQP clients, or Azure Event Hubs SDKs, without changing public access for other workspaces.

The setup requires Fabric capacity using an F SKU; Trial and Premium P SKU capacities are unsupported. Private-link-enabled eventstreams do not support Eventhouse direct ingestion mode, Activator, or custom endpoint destinations, so teams must check their downstream design before adopting the configuration.

Local model testing exposes concurrency trade-offs

A tested 64 GB laptop held Qwen3.8-27B, its draft model, and the full 262,144-token context. Short code generation measured 44.7 tokens per second, while the final full-context profile measured 44.1 tokens per second. Across three hours of OpenCode traffic, the system averaged 27 decoded tokens per second per request and 41 W of GPU power.

In separate 70-minute agent runs, one-request concurrency completed 111 requests versus 74 with two concurrent requests, showing why concurrency needs workload-specific testing. OpenTelemetry sends metrics to Azure Monitor managed service for Prometheus and traces to Application Insights using Microsoft Entra ID authentication. The setup connects local inference measurements with centralized monitoring rather than relying on isolated generation benchmarks.

Other Machine Learning News

Microsoft Discovery app agents interpreted over 120 software projects and reference documents, helping compress literature review, experimental design, permitting preparation, and synthetic classifier development from several weeks into a few days. The citrus-pest monitoring work used a replicated Latin square design to balance sampling locations and periods and synthetic Asian citrus psyllid biometrics for an initial edge classifier, while scientists retained responsibility for biological plausibility, experimental decisions, and agent-assisted permitting documents.

On-premises data gateway 3000.338 includes Power Query Engine 2.159.229 and October 2026 Power BI Desktop compatibility, preserves Power Query settings during connection tests, improves service-start diagnostics, and adds preview Admin Connections REST APIs for tenant-wide connection management. Microsoft recommends non-production validation and production cluster upgrades one member at a time, and supports only the last six gateway releases.

Also relevant this week