Where to Put Your AI Layer: Lessons from Real Projects
LLM integrations fail not because APIs are hard, but because architects treat them like regular features.
I've bolted AI into enough production systems to know: the integration is trivial. The architecture is everything.
Last year, I integrated Claude into an ERPNext system for a client. Their first instinct? "Just add it to the form submission handler." Classic mistake. One slow API response and the entire request times out. Users refresh. Database writes duplicate. You're now firefighting at 3 AM.
The real problems I've seen:
AI should almost never live inside your synchronous request flow. Period. I moved that ERPNext integration to async workers (RabbitMQ in that case). Form submits, returns immediately, AI processes in the background, updates the record when done. Problem solved.
Second: token costs are invisible until they explode. I built a Telegram bot that generated summaries on every message. Looked good in demos. Cost $400/month in production. Now I implement aggressive caching and only trigger AI on specific actions. Test your budget assumptions early.
Third: LLMs hallucinate. I learned this the hard way with a customer support system. The AI would confidently generate false information about product specs. Now every response is grounded in retrieved data first—RAG pattern. AI only summarizes or interprets what's already verified.
What actually works:
Treat AI like any expensive external service. Queue it. Rate-limit it. Cache aggressively. Monitor costs like your rent depends on it (it might).
In my Node.js projects, I use Bull queues. Process AI tasks asynchronously. Log everything—tokens used, latency, cost per request. Integrate with monitoring so you see cost anomalies immediately.
For legacy system modernization, AI is often the entry point to replace outdated business logic. But don't let that justify bad architecture. Just because the LLM API accepts a request doesn't mean your synchronous handler should call it.
Start with one well-architected integration. Get it right. Then scale. Your future self will thank you when you're not rewriting error handling at midnight.