I built an Voice-enabled AI agent that runs 100% locally — AMA on what broke

Hey PH

I've spent the last few months building an autonomous AI agent completely solo — no team, no funding, just me trying to build something that actually does things instead of being another chatbot wrapper.

A few technical decisions I'd love to get this community's take on:

1. Built the entire agent loop from scratch — no LangChain, no AutoGen.
Mainly because debugging tool-routing failures through someone else's abstraction layer was more painful than writing my own. Curious how many of you building AI products have made the same call, vs leaning on existing frameworks — and whether you regret either direction.

2. Made the task scheduler survive the app closing entirely.
Instead of an in-process scheduler that dies the moment you close the app, I integrated directly with Windows Task Scheduler so tasks keep running in the true background. Wondering what patterns others here have used for persistence outside the app lifecycle — especially if you're building for desktop rather than cloud.

3. Going fully local for voice — not cloud TTS/STT.
Early tests on low-end hardware have honestly surprised me with how fast it runs. If you've tackled local voice pipelines, I'd love to compare notes on what held up and what didn't.

The product itself (Sidus-Agent — autonomous, voice-enabled, tools/plugins/connectors/scheduler, spawns sub-agents for parallel tasks) is live if anyone wants context, but I'm mainly here for the conversation on the hard parts of building this stuff, not the pitch.

What's been the hardest architectural decision in something you've built? Genuinely curious what tradeoffs other makers here have run into.