We treat our AI as a creative writer and script everything boring. The output quality jumped
We run an automated pipeline that drafts articles for our product. During our first test attempts we let the model do almost everything, including the mechanical bits: fixing dashes and quotes, tracking which stage each article was in, creating SEO json, naming image files etc. It worked, but the quality was uneven, sometimes the model would miss some things, and, of course, it was rather expensive in terms of tokens.
Then we changed our approach and started treating AI like a creative employee and providing it with "software" (python scripts) to help with any mechanical tasks that could be automated by code. The model only does the part that actually needs intelligence and creativity now.
And honestly, the result surprised me - quality went up, because we now let it focus on what it's actually good at. As a pleasant bonus - the costs of running the pipeline dropped too. One of the first things we scripted was stripping the em-dashes the model loves to add but my co-founder has an allergy for 🤣
How do you improve the quality of your AI results, especially automated pipelines?
Replies
This feels like one of those lessons that's obvious only after you have experienced it. I'm curious whether you discovered this through trial and error or if there was a specific failure that pushed you toward scripting.
@ella_reyes1 Sadly, yes, at first we tried to add more agents into the pipeline, stricter rules, but results became even worse. It turned out that AI similarly to humans needs room for creativity, so we decided to minimize everything else so it's less distracted. This is where scripting came in.
The em-dash joke made me laugh 😂. Besides punctuation, what's another surprisingly simple automation that saved a lot of tokens?
@rose_florean Ha, glad it landed 😂 A few boring ones that added up:
SEO and frontmatter: we pre-fill all the mechanical fields in code (slug, canonical URL, dates, image path), so the model only writes the title and description.
State tracking: each article's status sits in a small JSON file, so the model never burns tokens figuring out which stage it's in.
Image filenames: derived from the alt text by a script, not the model.
None of them clever on their own, but together they took a real chunk off each run.
Separating creativity from deterministic work sounds like a much cleaner architecture. Do you think this approach scales well as your content pipeline becomes more complex?
@joseph_parker3 So far, yes. The nice part is that when things get more complex, the complexity lands in the scripts, not the model. Code is cheap to run, easy to test, and predictable, so adding steps there doesn't blow up cost or quality. The model's job stays small and focused no matter how big the pipeline gets. The real ceiling we hit is token and rate limits on the model side, and keeping it away from mechanical work is exactly what buys us room there.
I think this mirrors how human teams work too. you dont ask your best writer to rename files all day. has this changed how you design prompts now?
@gerald_don yes, the prompts became much cleaner and easier to maintain too.
this matches what I've seen too, the model doing renaming/formatting/bookkeeping is where most of the wasted tokens and the flaky output go. splitting "creative decision" from "mechanical execution" and giving the mechanical half to a plain script instead of another prompt is the same lesson as good software architecture, just applied to a pipeline instead of a codebase. did you have to rewrite the scripts much as the model's output format drifted over time, or has it stayed stable
@omri_ben_shoham1 we manually reviewed the whole pipeline and created scripts specifically for each part that could be replaced by code. But yes, we had to do a lot of changes on the orchestration side to make it work.
In addition, we followed my friend's advice and reduced the amount of steps, focusing on the actual prompt instead - this turned out to be a great advice because it did improve the quality and allowed us to produce much more articles before we hit the same limits.
@artk Yes, this as well. Fewer steps, stronger prompt - it made more difference than we expected.
Same experience here. The unlock for us was realizing the model gets worse when you make it do bookkeeping. Curious, after you added the scripts, did you stop needing to constantly rewrite your prompts?