Why should the demo run on your file, not the vendor's?
Every tool in this category now does roughly the same thing on the surface: it receives a request (a tender, a security questionnaire, a due diligence questionnaire, the DDQ a customer sends before awarding you a contract), matches it against a library of answers, drafts a first version, and orchestrates the review. On a demo dataset chosen by the vendor, they all shine.
What sets them apart only shows up on a real file, with its ambiguous questions, requirements your library does not cover, and wording traps. That is where you see whether the tool produces a defensible answer or a plausible one. The evaluation rule fits in one sentence: bring one of your own files, and ask the seven questions below while watching the screen, not the brochure.
The seven questions, and the evidence to demand for each
| Question | What a drafting assistant answers | What a response agent answers | The evidence to request |
|---|---|---|---|
| 1. Where does this answer come from? | "From the knowledge base," nothing more | The document, page, and version, shown next to each answer | Click three random answers and open the cited source |
| 2. What does it do when the answer is not in my documents? | It drafts a plausible-sounding sentence anyway | It declares the gap, names the expert to consult, and recommends the action to close it before submission | Ask a question you know the library has no answer for |
| 3. Who checks, and is it the same AI that drafted it? | The same one, in "review" mode | A second, independent AI team that scores each answer from 0 to 100 and flags discrepancies | Ask to see the score of an answer and the reasoning behind it |
| 4. Where are my documents processed, and are they used to train a shared model? | "On a secure cloud" | The country and deployment mode, a written commitment not to use your data for the benefit of other customers, including in aggregate, including at model subcontractors | The contract clause, not the website page |
| 5. Who signs, and how do you know the signature is real? | A "validate" button | Assigned validators by domain, a named and traced signature, and a vigilance test that inserts deliberately wrong answers into the flow and alerts if one is approved | Ask how the tool detects rubber-stamp validation |
| 6. What is left with the vendor at the end of the contract? | "You can export your data" | Handover in a reusable format, dated deletion covering derivatives (indexes, representations, logs), backups, and subcontractors, with an auditable certificate | The reversibility clause, with the list of what gets deleted |
| 7. Who does it learn from? | From all its customers, including your competitors | From your documents and your corrections, which stay within your instance | Ask what a correction made by your expert changes, and for whom |
A vague answer to one of the seven is a piece of information. A vague answer to three of them is a decision.
What the seven questions measure, beyond features
Because features look alike and catch up with each other quickly. The seven questions target what does not catch up: the structure of the tool.
Question 1 is about traceability: without a document, page, and version next to each answer, nothing is verifiable, not by you, and not by the buyer who will read your response.
Question 2 is about the honesty of the system: a useful AI knows how to say what it does not know. The right answer does not stop at abstaining; it continues with a recommendation.
Question 3 is about the independence of verification. A model that reviews its own output does not correct its errors without outside input, and its performance sometimes degrades after that self-review (Huang et al., ICLR 2024). Verification must come from a mechanism separate from drafting. On the Optivalue.ai platform, which publishes this site, this mechanism has 5 layers, including 7 anti-hallucination checks, and an answer that fails one of them never reaches the validator.
Question 4 is about trade secrets: your library of answers is one, and part of its legal protection depends on the protective measures you take, under the trade secret law applicable in your market (in the United States, the Defend Trade Secrets Act; in the United Kingdom, the Trade Secrets (Enforcement, etc.) Regulations 2018). Where the processing happens, how the data is used, and any subcontracting to a model provider are all part of that.
Question 5 is about the value of the signature. A validation that approves everything is not a validation. The only way to know is to test it.
Question 6 is about the long term: a tool you cannot leave cleanly is a tool you never really chose.
Question 7 is about what sets you apart. A tool that learns from all its customers pulls their answers closer to one another; a tool that learns only from you does the opposite.
The concession that clarifies everything
If your need is a faster first draft, reviewed end to end by your own teams before every submission, a drafting assistant is enough, and it costs less. The seven questions above do not disqualify it; they describe it. Buy it with your eyes open.
The seven questions matter as soon as the response commits the company without every line being reviewed by an expert: when volume exceeds review capacity, when the request is a security questionnaire or a DDQ, when the signature at the bottom of the document belongs to an executive. A draft commits no one. Your response does.
Three questions that draw the line
If you only have time for three of the seven, ask these, in this order:
- Question 2, on the gap. It is the fastest to test and the most revealing: ask a question you know your library has no answer for, and watch what comes out.
- Question 4, on data processing. It is the one that commits you the most, and the one whose answer lives in a contract, not a demo. Ask for the clause.
- Question 5, on the signature. It is the one nobody asks, and the one that separates a tool built by people who have already been audited from a tool built by people who have already sold something.
Frequently asked questions
How much time should you plan for a serious evaluation? The time it takes to process a real file end to end, with your experts in the room, through to submission. A platform demo takes one meeting; an evaluation runs on a real file, and the file sets the duration.
Should you compare several tools on the same file? Yes, if you can. The same file, the same seven questions, the same evidence required. The differences show up on their own, and they rarely come down to speed.
What is a security certification shown by the vendor worth? It answers part of question 4, the part about hosting security. It does not answer where processing takes place, whether your data is used to train a model, or whether it is subcontracted to a model provider. Those three points are found in the contract.
Can a tool pass all seven questions and still be a bad fit? It can pass all seven and still not suit your volume, your portal formats, or your sector. The seven questions rule out what should not be bought; they do not choose for you among what can be.
Why is the vigilance-test question asked so rarely? Because it requires admitting that human validation can become a rubber stamp. Teams that have already been audited know this; the others find out afterward.
Ask the seven questions on your own file
Bring a real tender, a security questionnaire, or a DDQ. You will see the extraction coverage, the sources cited to the page, the score of each answer, and the gap analysis on your own document, not a prepared demo. And you can ask the seven questions on screen.
Sources cited
- J. Huang, X. Chen, S. Mishra, H. S. Zheng, A. W. Yu, X. Song, D. Zhou, "Large Language Models Cannot Self-Correct Reasoning Yet," ICLR 2024. https://arxiv.org/abs/2310.01798
- Trade secret law (in the United States, the Defend Trade Secrets Act of 2016; in the United Kingdom, the Trade Secrets (Enforcement, etc.) Regulations 2018); the applicable rule in each market should be verified.