Ollama’s v0.40.0-rc1 updates its MLX tokenizer implementation to match publisher tokenizer semantics, affecting local model users whose inputs depend on exact token splitting and encoding behavior.
According to the release notes, the changes cover pretokenizer-stage ordering, split behavior, Unicode boundaries, added-token normalization and ranked BPE merges. The implementation also handles empty added tokens and empty Metaspace input consistently.
The release candidate adds shared Go and Python reference cases based on published tokenizers. Those tests can pull missing models directly and fail when errors occur, while additional regressions cover configuration precedence, byte fallback and parallel encoding.
These are compatibility and validation changes in a release candidate, rather than a new model announcement. Source: Ollama v0.40.0-rc1 release notes
Definition. Tokenizer compatibility is the ability of an implementation to split and encode input the same way as a published tokenizer.
Key takeaways
- The release candidate addresses pretokenizer ordering, split behavior, Unicode boundaries, added-token normalization, and ranked BPE merges.
- It handles empty added tokens and empty Metaspace input consistently.
- Shared Go and Python reference cases are based on published tokenizers.
- Tests can pull missing models and fail when errors occur.
- Additional regressions cover configuration precedence, byte fallback, and parallel encoding.
- The update is focused on compatibility and validation rather than a new model announcement.
FAQ
What does Ollama v0.40.0-rc1 change for MLX tokenizers?
It updates the MLX tokenizer implementation to match publisher tokenizer semantics, including pretokenizer ordering, splits, Unicode boundaries, normalization, and BPE merges.
What validation was added in the release candidate?
Ollama added shared Go and Python reference cases based on published tokenizers, along with regressions for configuration precedence, byte fallback, and parallel encoding.
Is Ollama v0.40.0-rc1 a new model release?
No. The supplied release summary describes compatibility and validation changes rather than a new model announcement.
