Open Source LLMs Hit the Sweet Spot for Production
Smaller open source models are finally good enough for real work. Here's what changed in my stack.
A year ago, I was shipping Claude API calls for everything AI-related. Easy, reliable, expensive. Now? I'm running Qwen locally for most tasks and only reaching for Claude when I actually need it.
The shift happened gradually. Open source models got better. Quantization got better. But mostly, I stopped treating LLMs like magic and started treating them like tools with tradeoffs.
In my Seven Suite products, I've integrated smaller models for:
- Document classification (98% accuracy on client data)
- Email/ticket routing (faster than humans, catches patterns we miss)
- Code generation helpers (good enough for boilerplate, I still review)
What changed my mind: I benchmarked them against real project work, not marketing demos. The 27B Qwen models handle structured tasks way better than their size suggests. Why? Because my prompts got tighter. I stopped asking the model to "be helpful" and started asking it to "extract the user_id from this log line."
The business case is obvious. A 27B model running on my own hardware costs nothing after the first inference. No API rate limits. No vendor lock-in. No audit trail sent to some cloud somewhere. That matters when you're handling client data.
For Seven CMS users who want AI features? I'm building this as an optional module. Run it locally, or use your own API key. Free core, paid support. Same philosophy.
The Hacker News thread nailed the real issue though: these benchmarks don't matter unless they work on YOUR data with YOUR latency requirements. A 27B model that takes 3 seconds per token is useless in a chat interface. But it's perfect for async batch processing.
I'm not ditching Claude entirely. Some problems still need the top tier. But the sweet spot? That's where the action is now. Smaller, open, deployable.
That's where I'm betting.