Challenging Tokenization: An LLM Experiment Without a Tokenizer
Most LLMs assume language must be broken into tokens (BPE, WordPiece) before computation. The tokenizer is a fundamental component.
While building a new experimental framework called NINMENI, we asked: What if tokenization isn't the only foundation?
A Different Starting Point
NINMENI explores a different primitive: 1 Character = 1 ID.
Character models aren't new, but what happens next is. Instead of treating characters as smaller tokens, the framework treats the character space as a native substrate. Meaning formation emerges from runtime interactions, not a tokenizer.
Complexity shifts:
Traditional: Complexity lives in segmentation algorithms.
NINMENI: Complexity moves to semantic organization.
Why This Matters
Traditional: Text -> Tokenizer -> Token IDs -> Model
NINMENI: Language -> Character Space -> Native Registry -> Meaning Units -> Model
The registry isn't just a Unicode list; it is a deterministic representation of the language space itself.
Engineering Lesson
Initially, we manually maintained a registry of thousands of characters, patching it whenever one was missing. This was the wrong abstraction.
The better approach: generate it from a specification.
Pipeline: Language Spec -> Writing Systems -> Relationships -> Unicode Coverage -> Generator -> Native Registry
Now, the registry is reproducible and deterministic. Adding a language means extending the spec, not editing thousands of entries.
Production Before Theory
We decoupled three distinct concerns:
Paradigm (conceptual rules)
Mechanism (operators)
Runtime (production infrastructure)
This allows the runtime to evolve without rewriting the conceptual model when implementation details change.
Why Share This?
This is an engineering experiment: Can an LLM use a different primitive while remaining practical for production?
If you designed an LLM from scratch today, which assumption would you challenge first?
Show Code
Replies