Most tool qualification work we are brought into starts the same way. A team has read ISO 26262-8 clause 11, concluded that their code generator needs qualification, and started writing a validation suite. Some months later the suite is large, green, and does not convince an assessor.
The suite was never the hard part.
What the standard actually asks for
Clause 11 asks a narrower question than teams assume. Not “is this tool correct” — that is not answerable for a compiler — but: if this tool malfunctions in the ways it plausibly can, will the resulting error reach the safety-related output, and would your development process catch it?
That question is answered by two determinations before any test is written. Tool impact, which asks whether a malfunction can introduce or fail to detect an error in the safety-related item. Tool error detection, which asks how confident you are that your existing process would catch it. Together they give a Tool Confidence Level, and the TCL decides how much you owe.
Teams that skip straight to testing skip the step that determines how much testing they needed. Frequently the answer was less than they built, and occasionally it was something testing cannot supply at all.
Use cases, not tools
The second recurring mistake is qualifying a tool rather than a use case.
“We qualified the static analyser” is not a claim that survives review. The analyser is qualified for a specific version, a specific rule configuration, a specific input language subset, and a specific way the project consumes its output. Change the rule set and the argument does not automatically travel with it.
This matters practically because it is what makes qualification expensive to maintain. A qualification argument scoped to a use case can be re-evaluated when the use case changes. One scoped to “the tool” has to be redone every time anything moves, and in practice it is not redone — it is quietly assumed to still hold.
Where the effort actually goes
On the engagements we have run, the distribution is roughly stable:
- Determining and justifying TCL. Short, and it decides everything downstream.
- Writing down the tool use case precisely. Version, configuration, inputs, how output is consumed, what the project does with a finding. This is the artefact that is almost always missing.
- Identifying credible malfunctions. Not hypothetical ones — the ones the vendor’s own release notes and issue tracker demonstrate.
- Validation suite. Real work, but bounded once the three above are done.
- Assembling the evidence. Cheap if the pipeline produced it, expensive if someone is reconstructing it from build logs the week before assessment.
The last line is where projects lose time they did not budget. Evidence that was generated as a by-product of normal development costs nothing to present. Evidence assembled retroactively costs weeks and is weaker, because nobody can show it corresponds to the build that actually shipped.
A shorter version
If you are about to start qualification, write the tool use case down first, in one page, and get someone who did not write it to read it back to you. If they cannot state which version, which configuration, and what happens to the output, you are not ready to write tests — and the tests you write will not answer the question you are going to be asked.