Guest: Hamel Husain

AI products and problems discussed:

• GitHub Copilot


• Forecasting AirBnB Guest Growth
  - NurtureBoss

Resources & Links

• Hamel’s blog on AI evals


• AI Evals for Engineers and PMs course on Maven (Get 35% off with this affiliate link)

Chapters:
00:00 Introduction to Hamel Hussein
00:34 Challenges in AI Consulting
02:00 Machine Learning Fundamentals
04:47 Debugging Machine Learning Models
05:00 Case Study: Airbnb's Guest Growth
08:51 Understanding Machine Learning Models
18:35 Introduction to Nurture Boss
25:40 Building AI Products with Synthetic Data
41:20 Connecting Machine Learning to Error Analysis
42:28 Real-World Example: Text Message Errors
44:15 Prioritizing and Documenting Errors
45:59 Continuous Improvement and Iteration
58:08 Using Synthetic Data for Evaluation
01:08:42 Avoiding Overfitting in Evaluations
01:19:28 Practical Tips for Error Analysis
01:25:10 Final Thoughts and Resources