This is a rewrite of a project of mine that used LexRank (https://arxiv.org/abs/1109.2128) to summarize long texts. Initially as an API it became a command line that I used to index data into search engines and general data manipulation. With the advent of LLM many tasks that could be done locally with machine learning algorithms were delegated to models, increasing cost and data leak risk. I've expanded tl;dt with TextRank and other protections I normally used to make it safe for people to use agentic coding and agents in general to build workflows including Unicode Confusables filtering, prompt injection detection, HTML to Markdown conversion and sanitization, along with better control on output size to maintain proper context. Also added a simple skill to support agentic coding. It sits close to rtk and caveman in terms of token saving but it focus on keeping the context of the summarized input intact. Cheers !

tldt — Too Long, Didn't Tokenize

Protect your data and agents while saving tokens

Protect your data and agents while saving tokens