Lightspeed search built for devs (Interview)
Digest
This podcast explores SearchCraft, an AI-native search engine built using Rust, highlighting its advantages over existing solutions and its focus on developer experience. The discussion includes interviews with Dom McKinnon, founder of SearchCraft, and Damien Schingelman from AuthZero. Key topics covered include the impact of AI on search, the role of Large Language Models (LLMs), the ethical considerations of using copyrighted material in AI training, and the potential effects of AI on software development, including the future of open-source and the evolving definition of "good code." The podcast also delves into the technical aspects of SearchCraft, such as its use of Rust, its embedded database (sled), and its approach to developer experience (DX). SearchCraft's features, including its REST API, SDKs, and user-friendly dashboard (Vectron), are detailed, along with future development plans. The podcast concludes with reflections on the future of search, AI, and multi-agent software.
Outlines
Introduction to SearchCraft and AI's Impact on Search
Introduction to SearchCraft, an AI-native search engine built in Rust, discussing its features, advantages, and the broader impact of AI on modern search. Includes discussion of AuthZero's approach to AI and identity security.
AuthZero, AI, and Identity Security
Damien Schingelman from AuthZero discusses AuthZero's preparations for the future of AI agents and the crucial role of security and identity in AI applications.
Deep Dive into SearchCraft's Architecture and Development
A long-time listener interviews Don McKinnon, delving into the technical details of SearchCraft, including its use of Rust, its storage engine (sled), and its developer experience.
Faster Builds, SearchCraft's Differentiation, and Developer Experience
Discussion on faster builds with Depot and how SearchCraft differentiates itself through developer experience and simplicity. Includes discussion of the challenges and rewards of building a search engine.
LLMs, SearchCraft's Adaptation, and Keyword Search Enhancement
The impact of LLMs on search and how SearchCraft is adapting by combining keyword search with natural language processing capabilities.
Conclusion: Future of Search, AI, and Ethical Considerations
Final thoughts on the future of search, AI, and the ethical considerations of using copyrighted material in AI training data.
AI's Impact on Software Development and the Future of "Good Code"
Discussion on how AI is changing software development, potentially reducing the need for junior developers and altering coding practices. Concerns about the future of open source and the evolving definition of "good code" are raised.
SearchCraft: Features, Future Plans, and Platform Integrations
Detailed overview of SearchCraft's features, including its REST API, SDKs, and user-friendly dashboard (Vectron). Future plans include expanding platform integrations and improving documentation.
SearchCraft Branding and AI-Generated Imagery
A brief discussion about SearchCraft's branding and website imagery, which utilizes AI-generated images with a science fiction theme. The process of creating these images, involving human curation and collaboration with AI tools, is explained.
Keywords
AI-native search engine
A search engine designed from the ground up to leverage the capabilities of artificial intelligence.
Rust programming language
A systems programming language used in SearchCraft for its efficiency and memory safety.
Levenshtein distance
A metric for measuring string similarity, used in SearchCraft for typo tolerance.
Developer experience (DX)
The overall experience a developer has when using a software tool or platform.
Large Language Models (LLMs)
AI models trained on massive datasets, impacting the future of search.
Agentic AI
AI systems that exhibit goal-oriented behavior.
AI-powered code generation
The use of AI to automatically generate code.
SearchCraft
A content-based search platform with a REST API, SDKs, and a user-friendly dashboard (Vectron).
Vectron
SearchCraft's user dashboard for managing search indexes.
Open Source Software (OSS)
The impact of AI on the future of open-source software.
Q&A
What are the key advantages of SearchCraft compared to existing search engine solutions?
SearchCraft prioritizes developer experience, offering a simpler, faster, and more efficient solution.
How is SearchCraft adapting to the rise of Large Language Models (LLMs)?
SearchCraft is integrating LLMs to enhance its semantic understanding of user queries.
What are the ethical considerations surrounding the use of copyrighted material in training AI models?
The use of copyrighted material raises complex legal and ethical questions; transformative use is a key consideration.
What is the future of search in the age of AI?
The future of search will likely involve a combination of keyword-based search and natural language processing.
How will AI impact the future of software development jobs?
AI-powered code generation may reduce the demand for junior developers, but increase the need for senior developers.
What is the current definition of "good code," and how is it changing with the rise of AI?
Traditionally, "good code" emphasized readability and maintainability; now, functionality and speed of generation are becoming more important.
How does SearchCraft address the challenges of search implementation for non-technical users?
SearchCraft provides a user-friendly dashboard (Vectron) for non-developers to manage search indexes.
What are SearchCraft's future development plans?
SearchCraft plans to expand its platform integrations, add support for more programming languages, and improve its documentation.
Show Notes
We talk with Don MacKinnon, Co-founder and CTO of Searchcraft—a lightspeed search engine built in Rust. We dig into the future of search, how it blends vector embeddings with classic ranking, and what it takes to build developer-friendly, production-grade search from the ground up.
Changelog++ members save 6 minutes on this episode because they made the ads disappear. Join today!
Sponsors:
- Auth0 – The identity infrastructure for the age of AI. Built by developers, for developers—Auth0 helps you secure users, agents, and third-party access across modern AI workflows. Token vaulting, fine-grained authorization, and standards-based auth, all in one platform.
Start building at Auth0.com/ai - Depot – 10x faster builds? Yes please. Build faster. Waste less time. Accelerate Docker image builds, and GitHub Actions workflows. Easily integrate with your existing CI provider and dev workflows to save hours of build time.
- Outshift by Cisco – AGNTCY is an open source collective building the Internet of Agents. It’s a collaboration layer where AI agents can communicate, discover each other, and work across frameworks. For developers, this means standardized agent discovery tools, seamless protocols for inter-agent communication, and modular components to compose and scale multi-agent workflows.
Featuring:
- Don MacKinnon –
- Adam Stacoviak – Website, GitHub, LinkedIn, Mastodon, X
- Jerod Santo – Website, GitHub, LinkedIn, Mastodon, X
Show Notes:
Something missing or broken? PRs welcome!
Table of contents
United States
