You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When using CPAchecker, or any other of our tools that utilize user-input, for example configurations, it might happen that you make mistakes when writing your command-line/options etc. I for example often write --spec memsafety in CPAchecker and get reminded that this spec does not exist. However, the error tells me nothing about available specifications, but just tells me we don't support this one. Therefore, I propose to improve this error message by providing suggestions based on the list of arguments that we allow.
An example based on my case above:
No property with the name 'memsafety' exists. Did you mean 'memorysafety'?
To do this, we should provide some algorithm for the distance between strings.
One possibility (that is easy to implement) is N-gram Jaccard similarity. It splits input strings into overlapping chunks (think memsafety and memorysafety -> (mem, safety) It returns a similarity score == 1.0 for equal, and > 1.0 for unequal input, that grows the more unequal they get.
I think it is a good idea to start with this algorithm and see whether this solves our use case.
More advanced solutions:
The normalized Levenshtein similarity algorithm also provides a easy score based similarity interpretation, for example:
Score
Meaning
1.0
Identical
0.8–0.99
Very similar
0.6–0.79
Moderately similar
< 0.6
Probably different
(Internal thresholds depend on string length etc.)
It should not be used for larger sentences due to its possible run time, but our use cases should be fine.
Also, there are solutions that improve certain characteristics further, for example Damerau-Levenshtein has better performance for swapped adjacent letters.
Should these be to expensive (which i doubt), we can fall back to other solutions.
A possible workflow could look like this:
add a suitable algorithm returning the similarity of two strings, so that we can find strings that are close to each other, used to e.g. detect typos in input vs. known strings based on how close they are. We can try the normalized Levenshtein similarity algorithm first, as it is known to work well for our cases (option names, singular words).
add a new method that uses this algorithm, and takes a input string stringToCompareTo, as well as a set of possible values, and a distance threshold and returns a possibly empty list of strings from the input set ordered by closeness to the stringToCompareTo in a descending fashion up to the threshold (the first string in the returned list is closest to stringToCompareTo)
add tests based on options used in our projects with common typos or alternative writing patterns. E.g. there is no property memsafety, as it is called memorysafety. This can also be used to determine which distance values we want to accept for our use case. (Note: examples for these values should then also be documented in the JavaDoc of the algorithm). This should also include tests for edge-cases, null etc.
investigate whether we can integrate this directly into our Configuration API with the goal of returning a set of close and existing options should a option not exist. Here we need to be careful not to add more reflection magic etc. Simpler is better. A good point to start this investigation would be the inject() method here.
potentially add this similarity check directly to our Configuration package automatically. We also want to make this optional, so that it can be used if wanted. Also, we don't want to enforce how this is reported. Users need to be able to implement this for themselves, or choose our predefined way. (We could try to find a common way of using our logging system after we established that this works, to not re-use the same code in our projects over and over again.)
When using CPAchecker, or any other of our tools that utilize user-input, for example configurations, it might happen that you make mistakes when writing your command-line/options etc. I for example often write
--spec memsafetyin CPAchecker and get reminded that this spec does not exist. However, the error tells me nothing about available specifications, but just tells me we don't support this one. Therefore, I propose to improve this error message by providing suggestions based on the list of arguments that we allow.An example based on my case above:
No property with the name 'memsafety' exists. Did you mean 'memorysafety'?
To do this, we should provide some algorithm for the distance between strings.
One possibility (that is easy to implement) is N-gram Jaccard similarity. It splits input strings into overlapping chunks (think
memsafetyandmemorysafety-> (mem,safety) It returns a similarity score == 1.0 for equal, and > 1.0 for unequal input, that grows the more unequal they get.I think it is a good idea to start with this algorithm and see whether this solves our use case.
More advanced solutions:
The normalized Levenshtein similarity algorithm also provides a easy score based similarity interpretation, for example:
1.00.8–0.990.6–0.79< 0.6(Internal thresholds depend on string length etc.)
It should not be used for larger sentences due to its possible run time, but our use cases should be fine.
Also, there are solutions that improve certain characteristics further, for example Damerau-Levenshtein has better performance for swapped adjacent letters.
Should these be to expensive (which i doubt), we can fall back to other solutions.
A possible workflow could look like this:
stringToCompareTo, as well as a set of possible values, and a distance threshold and returns a possibly empty list of strings from the input set ordered by closeness to thestringToCompareToin a descending fashion up to the threshold (the first string in the returned list is closest tostringToCompareTo)memsafety, as it is calledmemorysafety. This can also be used to determine which distance values we want to accept for our use case. (Note: examples for these values should then also be documented in the JavaDoc of the algorithm). This should also include tests for edge-cases,nulletc.inject()method here.Configurationpackage automatically. We also want to make this optional, so that it can be used if wanted. Also, we don't want to enforce how this is reported. Users need to be able to implement this for themselves, or choose our predefined way. (We could try to find a common way of using our logging system after we established that this works, to not re-use the same code in our projects over and over again.)(moved partially from a CPAchecker issue to here.)