Test against a reference
The text went in. What came out?
In the Rust port of SysON, the software could recognize the grammar of a SysML file and still refuse to turn it into a model. Even when the importer succeeded, I needed to know whether a part ended up inside the right assembly and whether its type pointed to the right definition.
Give both tools the same input
The comparison in the SysON project takes the model produced by a released version of SysON as its reference. It compares the model structure, allowing the tools to assign different identifiers to the same parts. A link from a part to the wrong definition still counts as a difference.
Keep the refusals in the count
In the November 18 replay, the Rust importer matched the reference structure on 12 of 62 cases. On one, both produced a model but disagreed about its structure. On the other 49, the reference imported the text and Rust refused it. Counting only the cases that produced two models would hide most of the work left to do.
Try a wheel and a comment
Open the typed wheel and comment examples. For each, sketch the package and the definitions you expect to find inside it. Then look up p01 and p05 in the earlier case-by-case run. Which case let you compare two models? For the refusal, suggest a smaller input that would help locate the problem.
What a match can tell us
A match tells me that the tools agreed on the structure we compared for that input. Deciding whether the reference itself handled a disputed case correctly takes a separate check against the language rules.