AI Features Need Evidence, Not Just Tests

I shipped an AI feature last month. Looked solid in staging. Tests passed. Then a customer asked: "Why did it give me a completely wrong invoice calculation?"

I had no idea.

Turns out my checklist was theater. I had:

• ✓ Unit tests
• ✓ Integration tests
• ✓ Code review approved
• ✓ No linting errors

But I had zero evidence the AI was actually making sensible decisions. No token logs. No prompt versions tracked. No before/after comparisons. Nothing.

When you're building with Claude or any LLM, tests aren't enough. You need a paper trail.

Here's what I do now:

Log everything about the prompt. Version it. Tag it with the feature flag. When something breaks, I can actually see which prompt was running. Sounds obvious, but I didn't do it.

Test with real data, not synthetic. Your test data is always too clean. Take 50 actual customer records and run them through the feature. Save the inputs and outputs. Compare them manually.

Build a simple audit table. Just store: what went in, what came out, was it flagged as suspicious, did a human verify it. That's your production evidence. When a customer says "this is wrong," you have context.

Keep prompt iteration in git. Same discipline as code. If I change the system prompt, I commit it with a message. I can bisect failures.

I'm building this into Seven Suite now. ERPNext integrations especially need this because financial data isn't forgiving. One wrong AI calculation tanks trust.

The dangerous thing about AI features is they fail silently. A bug in code crashes loud. A hallucination in an invoice calculation just... happens. The customer notices later. By then you're in postmortem mode.

So yeah, get your checkmarks. But before you deploy, ask yourself: could I explain this feature's decision to a customer? Do I have the logs to back it up?

If the answer is no, it's not ready.