Local LLMs in Production: Privacy vs Practicality

Everyone's excited about local LLMs now. "Your data never leaves your machine," they say. Privacy theater? Maybe. But I've actually shipped this stuff, and the reality is messier than the marketing.

I integrated Qwen and Llama into a few projects last year. One was an ERPNext document processor. Sounded perfect: keep everything internal, no API costs, no latency waiting for external services.

First problem: hardware. A 7B parameter model needs ~15GB VRAM minimum for reasonable inference. A 13B? You're looking at 24GB+. My client's server had 32GB total. That's tight when you're also running PostgreSQL, Redis, and the actual ERP. One spike and everything chokes.

Second: speed. Local inference is slow. We're talking 50-100 tokens per second on decent hardware. For batch processing documents? Fine. For real-time chat or live suggestions? Your users hate it.

Third: maintenance burden. Ollama works, but it's another service to monitor, upgrade, and troubleshoot. When inference crashes at 2 AM, that's on you now. Not OpenAI's problem anymore.

That said, I still use local models where it makes sense:

• Batch processing: Document classification, invoice parsing. Run it overnight, who cares about latency.
• Sensitive data: Healthcare, financial records. Sometimes compliance requires no external calls.
• Cost at scale: Processing millions of items? Local wins economically.
• Offline capability: Field teams with spotty connectivity need it.

But for most business automation? I route through Claude or GPT-4o. The cost-per-inference is lower than my server hardware, they're better models, and I sleep better knowing someone else handles the infrastructure.

The "privacy" angle is real, but it's not binary. It's a tradeoff. You're trading convenience and performance for data locality. That's a valid choice for some use cases. Just don't let the marketing make the decision for you.

If you're building something that touches sensitive data at scale, test local inference. But be honest about the operational cost. Privacy matters, but so does reliability.