What hidden server expenses come up when moving from API keys to open-source models?

As my app traffic grows keeping up with open AI API costs is becoming a huge monthly expense. I've been eyeing self-hosted setups for Llama to cap my spending but I keep hearing mixed stories. Running cloud GPUs 24/7 means paying even when traffic is low and managing server maintenance takes away time I could spend building features. Plus if output quality drops users might notice right away. Have you swapped paid APIs for hosted open-source models and did it lower your monthly run rate or just add headache?