Guests:

• SallyAnn DeLucia, Director of Product, Arize


• Jack Zhou, Staff Engineer, Arize

In this episode, we cover:

• What tracing, observability, and evals really mean in GenAI applications


• How Arize used its own platform to build Alyx, its AI agent


• The role of customer success engineers in surfacing repeatable workflows


• Why early prototyping looked like messy notebooks and hacked-together local apps


• How dogfooding shaped Alyx’s evolution and built confidence for launch


• Why evals start messy, and how Arize layered evals across tool calls, sessions, and system-level decisions


• The importance of cross-functional, boundary-spanning teams in building AI products


• What’s next for Alyx: moving from “on rails” workflows to more autonomous, agentic planning loops

Resources & Links

• Arize AI — Sign up for a free account and try Alex


• Arize Blog — Lessons learned from building AI products


• Maven AI Evals Course — The course Teresa took to learn about evals (Get 35% off with Teresa’s affiliate link)


• Cursor — The AI-powered code editor used by the Arize engineering team


• DataDog — For understanding application traces


• OpenAI GPT Models — GPT-3.5, GPT-4, and newer models used in early and current versions of Alex


• Jupyter Notebooks — A tool for combining code, data, and notes, used in Arise’s prototyping


• Axial Coding Method by Hamel Husain — A framework for analyzing data and designing evals

Chapters:
00:00 Introduction to Sally Ann and Jack
01:08 Overview of Arize.ai and Its Core Components
01:44 Deep Dive into Tracing, Observability, and Evals
03:56 Introduction to Alyx: Arize's AI Agent
04:15 The Genesis and Evolution of Alyx
08:51 Challenges and Solutions in Building Alyx
24:33 Prototyping and Early Development of Alyx
26:22 Exploring the Power of Coding Notebooks
26:51 Early Experiments with Alyx
27:59 Challenges with Real Data
29:20 Internal Testing and Dogfooding
31:55 The Importance of Evals
35:16 Developing Custom Evals
43:09 Future Plans for Alyx
47:59 How to Get Started with Alyx