Launching today
PyTokenCalc
About Universal token counting for ANY LLM.
14 followers
About Universal token counting for ANY LLM.
14 followers
Universal token counting for ANY LLM. 20+ providers (cloud + local), custom models, pattern-based forward-compatibility, 99%+ accuracy. - Mullassery/PyTokenCalc
One thing that would make this way more useful for me is a CLI mode or simple Python one-liner so I can quickly count tokens from the terminal without writing a full script. Something like `pycount "text here"` or piping from stdin would be perfect for quick checks.
@saniye857589 - Upgrade. Its added to the latest version.
Finally a token counter that works across all my models without juggling different libraries. Tested it against a few cloud APIs and the numbers matched perfectly, way less hassle than I expected.
finally a tokenizer wrapper that actually works with the local models I run, not just openai. dropped it into my pipeline and the counts matched my hand checks within rounding.
Finally a token counter that works for local models and weird custom endpoints. Tested it against tiktoken and a couple of Ollama setups, numbers matched perfectly and the pattern fallback is genuinely clever for models without official tokenizers.
handy little lib, dropped it into a quick script and it just worked across openai and a local llama model without any extra config.