The Data Engineer's guide to optimizing Kubernetes, with Niels Claeys
Description
Niels Claeys shares how his team at Dataminded built Conveyor, a data platform processing up to 1.5 million core hours monthly. He explains the specific optimizations they discovered through production experience, from scheduler changes that immediately reduce costs by 10-15% to achieving 97% spot instance usage without reliability issues.
You will learn:
Why the default Kubernetes scheduler wastes money on batch workloads and how switching from "least allocated" to "most allocated" scheduling enables faster scale-down and better resource utilization
How to achieve 97% spot instance adoption through strategic instance type diversification, region selection, and Spark-specific techniques
Node pool design principles that balance Kubernetes overhead with workload efficiency
Platform-specific gotchas like AWS cross-AZ data transfer costs that can spike bills unexpectedly
Sponsor
This episode is brought to you by Testkube—where teams run millions of performance tests in real Kubernetes infrastructure. From air-gapped environments to massive scale deployments, orchestrate every testing tool in one platform. Check it out at testkube.io
More info
Find all the links and info for this episode here: https://ku.bz/hGRfkzDJW
Interested in sponsoring an episode? Learn more.
United States
