DiscoverPretrainedTokenizers now and in the future
Tokenizers now and in the future

Tokenizers now and in the future

Update: 2025-08-14
Share

Description

History of tokenization going back to 70s language modeling, modern tokenization approaches with BPE, and the future of token-free bitestreams

Comments 
00:00
00:00
x

0.5x

0.8x

1.0x

1.25x

1.5x

2.0x

3.0x

Sleep Timer

Off

End of Episode

5 Minutes

10 Minutes

15 Minutes

30 Minutes

45 Minutes

60 Minutes

120 Minutes

Tokenizers now and in the future

Tokenizers now and in the future

Pierce Freeman & Richard Diehl Martinez