Thoughts on NLP, life, and everything in between.
The tokenizer isn't a fixed black box — a plain regex runs before the tokens and quietly shapes how well a model handles your input. Read it, and you can get real quality gains at inference time without training anything.
Read →