The questions are gathered first
Development is driven by a set of realistic questions gathered with you. In this scenario I don't think you can say "ask whatever you like and it will answer".
Querying business data with AI: when the answer must be exact.
I am developing an application that lets you query business data in natural language, starting from imported delivery documents. It combines data search, language models and conventional code (in the jargon, a RAG system): the model helps interpret requests, while counts and totals are calculated by the database. The guiding requirement is precision: if you ask which products sold best over a period, the result must match the one in the management system. The project is currently under development and testing.
The figure must matchThe one in the management system, not just resemble it.
Under development and testingA growing skill, not an established specialisation: the technologies are changing quickly.
The perspective I write from
I started studying machine learning around 2020, followed by deep learning, generative AI and language models, through courses, documentation and experimentation: that is the foundation I work from. My direct experience on real projects is much more recent, and with large generative AI platforms in production it is still limited.
These observations bring that study together with nearly thirty years of real software development. I focus particularly on sustainability: a system must provide enough value to justify its development, verification and ongoing operation.
What it means for whoever commissions the project
A first version that answers plausibly is quick to build. A system that gives the right figure is a different job: the language model, on its own, does not guarantee exactness, and has to work alongside traditional code that calculates and checks. The effort grows far faster than the precision you gain.
Every step is verified
Every phase has its own check, from the start: you always know at which point a result stops being correct. Otherwise the error is discovered at the end, with no idea where it begins.
Better no answer than an invented one
When the question is unclear, the system asks for clarification; when a check does not pass, the request fails instead of returning a doubtful answer.
How it works, in brief
The language model steps in at three points at most; everything that is a number remains code.
- The code recognises in the question the names, codes and quantities that actually exist in the data.
- The most frequent questions follow fixed paths, without going through a model.
- For the others, the model proposes how to search; the code checks the proposal before executing it.
- Counts and sums are computed by the database; the model writes only the text of the search answers, citing data the system has verified.
What is never left to the model
Where I went wrong and what I learned
Four mistakes from this project, which is still under development and testing. The examples are constructed to illustrate the mistake; they are not real questions or data.
1. The question misread, silently
The mistake. Customer Rossi, in the chat on the company's website, asks: "How much have you delivered to me in 2025?". The program composes the request and sends it to the engine. If the model drops the customer's name, the system answers with the company's total: Rossi sees data that is not his. The figure is true and no check fires: it is not an error, it is a plausible result. This scenario is possible, not a current use of the project, but the risk does not depend on the type of data or on who is asking.
It applies inside the company too: an employee who asks for "my department's budget" and receives everyone's, or who sees the results of individual sales agents. It also happens in other ways: the model ignores a period or a product, or the program applies a default value even though the request specified one. A single word is enough: "orders in June 2025" and "orders up to June 2025" mean one month and a range. The model reads it correctly most of the time, and sometimes not, and a rare error does not show in a demonstration.
How you notice. Who may see what is enforced by the code, starting from who is asking, and does not depend on how the model reads the question: an instruction to the model can be bypassed. Then I ask the model not only what it understood but which words of the question it used, and where. The code reads the question again: if "up to" came before "June 2025", the model left it out and the code corrects the interpretation. The words of whoever asks override what the model proposes.
There is also the opposite error: the model refuses the question because it judges it not unambiguous enough. It exposes no data, but whoever asked is left without an answer, and I count it as an error.
What to ask whoever is developing it. What stops a user from seeing data that is not theirs: the way the model reads the question, or a check that does not depend on it? What does the system do when part of the question is not explained by what it has interpreted?
Security: which data the system can reach
2. Precision is measured, not felt
The mistake. Judging the system by a few questions tried by hand. It works on the questions you remember and gets wrong the ones nobody tried.
How you notice. A set of reference questions, each with the expected answer written and approved by a person: from a few dozen to about 150, depending on whether a single change is being checked or the whole procedure is being qualified. At each run I count the wrong interpretations (period, customer or product ignored, a default value in place of the one given), the refusals for ambiguity and the differences between two runs of the same question: even with the same prompt, the same question can give different answers. I tune on one part of the questions and verify on another, and I never write the words of a test question into the code.
What to ask whoever is developing it. Who wrote and who approved the expected answers? How many are there, and what happens to the questions the system does not answer? The expected answers have to be written and approved by people in the company: that is time to allow for.
3. One model at a time
The mistake. Trying several models at once. Besides Qwen3.8-27B I tried Gemma 4 in its 26B and 31B versions: they produced more wrong interpretations, of the same kinds seen above. They would probably each need a prompt tuned for that model. With three models I could not tell the model's fault from the code's.
How you notice. I stopped at one model, until I had a baseline that works and can be measured. Trying the others to see the differences is the next step, not yet done. Changing model is not a swap: it is a new piece of testing work.
What to ask whoever is developing it. How many models has the system been tried on? What changes, and what stays the same, if the model changes?
Changing model: only with tests
4. One rule per problem, not a patch per case
The mistake. A review finds a problem and I fix it. The next review finds another one, different but in the same area of the procedure: often another aspect of the same problem, sometimes a consequence of the previous fix. Each fix, taken alone, is right; but going one point at a time does not end the series, because nothing addresses the rule the whole area depends on. Constructed example: I add the customer filter to the search; the next review finds that, when the filter cannot be applied, the search falls back to a default value; the one after finds the same weak point in the export.
How you notice. When two reviews find different problems in the same area, I do not fix the third point. For each problem I adopt a single rule, written to block when in doubt (fail-closed): if the system cannot show that an answer is allowed, it does not give it. I check the rule with a test generated over many cases, not a few hand-written examples. A new case is added to the generated cases, not handled as a separate fix.
What to ask whoever is developing it. When a review finds another problem in the same area, what changes: another fix or a rule? Does the test cover the possible cases or only the ones already seen? When in doubt, does the system answer or stop?
What it costs to run
The method also matters because the result depends heavily on the model used, and neither option comes free.
Online services
In tests for the project described, online services outperform the local configuration; these aspects still need consideration:
- costs depend on service consumption and can grow quickly under heavy load;
- privacy depends on what reaches the model: contracts with processing in Europe and no data retention are available. In this project, in any case, the deterministic code extracts the sensitive part, and the model never sees it.
Open model on your own server
An open model on a GPU server avoids a provider's per-call fee and keeps the data in-house. It may pay off with high, steady volumes, or be needed when data cannot be sent to external services. It still has costs and limits to consider:
- hardware has purchase and running costs, to be checked for the chosen configuration;
- in the configuration I tested (a quantised Qwen3.8-27B, on 32 GB), it handles 2–3 concurrent requests;
- a request takes 10 seconds or more, and execution has to be optimised;
- in the smaller models I have tested, precision and Italian-language handling are limitations to check against the specific task.
In my direct experience, all this calls for very precise analysis and development, with meticulous attention to the client's needs: the opposite of what you often read online.
And systems that call the model many times?
Given the constraints of the local configuration described above, I consider workflows that call the model many times for each request demanding to sustain. I see more scope in high-value requests at limited volumes, or in faster services whose costs fit the work performed. Each call consumes tokens, and how many are needed for a correct answer can only be found by measuring (the cost of the whole service). This judgement concerns the configuration I tested and should be revisited if models, hardware or requirements change: cheaper models and better-bounded workflows can change the economics.
The three routes compared: costs and continuity
The full flow, for those who want the technical detail
The language model steps in at three points at most; counts, sums, filter application and citations remain deterministic code.
Codedeterministic stepModelcall to a language model
Preparation nothing is executed yet
- 01QuestionCode
The natural-language question, with any explicit filters from whoever asks it.
- 02Value recognitionCode
Looks in the question for values that actually exist in the data, such as names and codes, and for quantity mentions.
- 03Fast pathsCode
The most frequent questions are recognised by deterministic rules, without calling a model.
- 04Coverage checkCode
Is the whole question explained? If part of it is left uncovered, the fast path is discarded rather than giving a partial answer.
- 05ClassificationModel
Only when no fast path is enough: the model says whether the question asks for an analysis or a search.
-
Depending on the classification, one of two paths:
06aCompilationCodeFor analyses, a deterministic compiler turns the question into a plan.
06bPlanningModelFor searches, the model proposes one to three steps as typed JSON. The code validates them: schema, the asker's filters, terms not found in the question.
- 07Value resolutionCode
Every value in the plan is checked against the real data. If it is unknown or ambiguous, a clarification is requested.
- 08Prepared planCode
If the data changed in the meantime, preparation is repeated: an out-of-date plan is never executed.
Execution
-
Depending on the kind of plan, one of two paths:
09aAnalytics and aggregatesCodeCounts, sums and analyses are computed by the database; text and citations are generated by code. No model involved.
09bSearchCodeExact search in the database or semantic search on a vector index, then confirmed against the active data. At most 40 pieces of evidence.
- 10Answer draftingModel
Searches only: the model writes the answer, citing the identifiers of the evidence it received.
- 11Citation checkCode
The server checks every identifier and rebuilds the citations. An unknown reference makes the request fail.
- 12AnswerCode
Text, status, executed plan and citations, with notes on the limits applied.
The other exits
- Clarification or unsupported question. When the plan cannot be built with certainty: a deterministic message, never an invented answer.
- Error. If validation fails, if the model cites evidence that doesn't exist, or if the data keeps changing: the request fails instead of returning a doubtful answer.
Supporting sources
These references support the general technical criteria. Measurements on my hardware describe that configuration and do not automatically apply to others.
- Microsoft — evaluating RAG systems (opens in a new tab): distinguishes retrieval relevance, grounding and final answer quality.
- Google Cloud — agentic architectures (opens in a new tab): describes latency, cost and complexity trade-offs in workflows with multiple agents and model calls.
- Evalita-LLM — Italian-language benchmark (opens in a new tab): provides Italian tasks for comparing models; it does not replace testing the intended use case.
- Databricks — adoption of open models (opens in a new tab): aggregated customer data documents the use of open models; the sample does not represent the entire market.
- Databricks — agentic system trends (opens in a new tab): reports growth in multi-agent workflows among its customers, without showing that they suit every use case.
- OWASP — Sensitive Information Disclosure (opens in a new tab): least privilege and the limits of instructions given to the model; it does not describe per-user filters in retrieval.
- Thinking Machines — nondeterminism in inference (opens in a new tab): at temperature 0 the same request can give different answers, measured on one model and one server; it does not describe my configuration.
- Sclar et al. — sensitivity to prompt format (opens in a new tab): different models react differently to format; a study of earlier models, not of my case.