DAIS 2025

The Databricks Data + AI Summit 2025, held in San Francisco from June 9–12, brought enterprise data leaders together to explore the latest in unified platforms and AI-native systems. This year’s focus shifted from fragmented toolsets to governed Lakehouse architectures that accelerate insight delivery and simplify AI deployment. 

My Fave Five!

From Code Completion to Autonomous SE Agents

In this workshop, the introduction of SWE-bench—a benchmark built from real GitHub issues—immediately stood out as a much-needed yardstick for measuring AI’s real-world development capabilities. Watching SWE-agent orchestrate issue triage in a live demo drove home how a simple prompt like “create a script to reproduce the error and execute it” can unlock powerful autonomous workflows, even in sprawling codebases. I appreciated the candid discussion around agent limitations—especially how locking an agent to rigid patterns can backfire under high task variance—underscoring the need for thoughtful sandboxing and trajectory analysis. From my own notes, the emphasis on function calling and handling environment errors reminded me that robust failure-recovery is as critical as clever prompt design. The session left me energized to experiment with lightweight, stable AI assistants in my daily engineering tasks, leveraging fine-tuning rather than over-engineering.

Scaling XGBoost With Spark Connect ML on Grace Blackwell

In this breakout, the team’s work on leveraging the Grace Blackwell chip for XGBoost’s out-of-core training truly impressed me—seeing how NVLink-C2C slashes data-transfer bottlenecks brings terabyte-scale boosting into reach on a single node. Their quantile-based DMatrix batching and external-memory strategies highlighted practical trade-offs: avoiding OOMs without sacrificing data locality remains an art, not just an API call. I appreciated the candid comparison of vertical vs. horizontal scaling—realizing that combining Spark Connect’s MLCommand interface with GPU-accelerated out-of-core training can simplify cluster orchestration for massive tabular workloads. From my notes, the emphasis on explainability via exact SHAP values reaffirmed why decision trees often win over opaque neural nets in regulated industries. After attending, I’m inspired to prototype a similar architecture for our in-house feature store, using QuantileDMatrix batching paired with fine-tuned Spark Connect pipelines.

No-Code Change in Your Python UDF for Arrow Optimization

As someone who often relies on standard Python UDFs in Spark for their simplicity, I found the ability to enable Arrow optimization without touching a line of code truly game-changing. The session clearly explained how toggling a configuration flag or UDF-level parameter brings you the same performance wins as Pandas UDFs—vectorized by NumPy—while preserving your existing API. It was eye-opening to learn about the 2 GB batch size limitation and how NumPy’s own vectorized reads can sometimes offset expected gains, emphasizing the importance of benchmarking on your own workloads. The practical tips on balancing simplicity versus performance—like when to fall back to Pandas UDFs or split large batches—resonated deeply with my experience optimizing ETL pipelines. I left with a concrete plan: audit my most performance-sensitive UDFs, flip the Arrow switch, and measure improvements before refactoring.

Goose – Open Source AI Agent

The Goose session introduced a refreshingly modular, MCP-based agent framework that seamlessly bridges language models with real-world tools—from JIRA to GitHub to shell scripts—without locking you into a single IDE. I was struck by its autonomous execution mode and robust feedback loop, where failed tool calls loop back as input rather than crashing the workflow, a design choice I’ve already earmarked for our own CI/CD automations. The context-revision strategy—sending only diffs to the LLM while presenting full changes to the user—strikes the perfect balance between token efficiency and transparency. Seeing both a CLI and a desktop app demo inspired me to prototype a Goose agent that auto-generates PRs from JIRA tickets in our sprint backlog.

Accelerate End-to-End Multi-Agents on Databricks and DSPy

This session crystallized how production-ready GenAI pipelines demand more than just an LLM—they need a fully integrated platform that spans ETL, governance, development, and deployment. Watching the Mosaic AI Agent Framework orchestrate multi-agent reasoning via DSPy demonstrated how chaining smaller, specialized models can outperform one monolithic LLM, both in speed and maintainability. The live Gradio demo—where agents dynamically selected the correct function call, fetched data via Unity Catalog, and iteratively refined answers—gave me a clear blueprint for building robust, data-intelligent chatbots in our own stack. I was particularly struck by how straightforward prompts have become now that tool-calling is baked in, reinforcing the mantra: simple prompts plus strong function hooks > complex prompt engineering. After this talk, I’m eager to prototype a multi-agent workflow in Databricks using DLT for data prep, DSPy for agent orchestration, and AI Gateway for seamless serving.

The Databricks Data + AI Summit 2025 made one truth crystal clear: the future of analytics lives at the intersection of a governed Lakehouse and agent-powered intelligence. If these highlights sparked ideas for your own stack, now’s the time to spin up a free edition and start prototyping! I’ll be experimenting right alongside you—and counting down to Summit 2026 to see how far we all push the Lakehouse frontier. Until then, happy building!