Explainability is an architecture decision, not a UI feature

The standard product development order is: build the recommendation feature, prove it works, then add explanation as a UI concern. This is wrong — not because it is lazy, but because the information you need to explain a decision no longer exists by the time the explanation feature is on the roadmap.

To explain why a system recommended pathway A over pathway B for a specific user, you need to know which signals drove the score, in what proportion, and why those signals were considered relevant for that user's profile. If the scoring function runs those calculations internally and returns only a float, that information is gone. You can describe the output. You cannot prove the process.

The LLM shortcut does not solve this. You can ask a model to explain why user X matched pathway Y based on their profile. It will produce a fluent, confident explanation. It will frequently be wrong. The model does not have access to the scoring function's internal state — it is reasoning from the input and output, not from the process. This is explanation as hallucination. It looks like explainability. It is not.

The correct order is to design the reason codes before the scoring algorithm. In AICA, every recommendation carries a reasons[] array that is built during scoring — strings like 'Strong match: prefers analytical and quantitative work' or 'Partial: pathway is on-site heavy but your preference is remote-first'. These are produced by the scoring function itself as it evaluates each signal. They are deterministic, auditable, and exist before any LLM is involved.

The LLM earns its place further down the flow. When a user clicks 'Why' does this fit me?' — not the surface explanation, but the deeper one — the model receives the structured reasons as input and turns them into richer language personalized to the user's context. It is amplifying reasoning that was already captured. It is not becoming the source of truth.

The cost argument reinforces the architectural one. Deterministic scoring runs for every user at zero inference cost. Generative explanation runs only when the user explicitly requests depth — and at that point, the model is working from evidence rather than improvising. Both components do what they are actually good at, and neither is carrying the weight of the other's responsibility.