Tokenizers now and in the future

Update: 2025-08-14

Description

History of tokenization going back to 70s language modeling, modern tokenization approaches with BPE, and the future of token-free bitestreams

Comments

In Channel

A deep dive on OpenAI Atlas

2025-10-3101:13:09

The browser wars are just getting started

2025-10-2957:57

Are we in an AI bubble?

2025-10-2401:15:19

LLMs can get brain rot too

2025-10-2301:00:34

AMD is back in the AI chipset race

2025-10-1758:27

The inaugural listener mailbag

2025-10-1455:46

California legislators come for LLMs

2025-10-0901:07:53

Move over TikTok - a new feed's in town

2025-10-0701:02:24

Gen z struggles to find coding jobs fr no cap

2025-10-0251:49

The power of ten million deadlifters

2025-09-3001:06:27

How countries are actually using AI

2025-09-2551:18

Your new raybans just got smarter

2025-09-2301:00:54

The claude code conspiracy

2025-09-1953:50

Will McTighe on selling through social media

2025-09-1601:10:57

Eating some mooncake

2025-09-1233:55

Training a 1 trillion parameter model

2025-09-0443:08

Nano banana is our favorite fruit

2025-09-0250:38

A breakdown of Genie 3's world model

2025-08-2857:10

Using chat in history class

2025-08-2139:52

AI is your new favorite songwriter

2025-08-1949:51

00:00

Tokenizers now and in the future

Pierce Freeman & Richard Diehl Martinez

#box-pro-ellipsis-176209698710848{-webkit-line-clamp:2;}Tokenizers now and in the future

Tokenizers now and in the future

Pierce Freeman & Richard Diehl Martinez

Tokenizers now and in the future