I am a big fan of TBCEditor [1] by Lasse Rautiainen, in case you have not heard about it, it is fork of SynEditthat diverged very significantly and includes support for code folding, fully JSON-based syntax highlighters, minimap and many other features.
Of course a code editor is a very complex piece of software, with many opportunities for bugs to hide. I have been helping by providing some Fuzz Testing [2] code and vectors.
Fuzz Testing, aka Monkey Testing for those that prefer the cuter, older terminology, is essentially a take on the Infinite monkey theorem [3] :
A thousand monkeys at a thousand typewriters will eventually type out the entire works of Shakespeare
But where the “works of Shakespeare” are a metaphor for the particular sequence of inputs that will cause a particular software to fail. This can also be seen as a form of Monte-Carlo testing.
Monkey Tester
A Monkey (pardon Fuzz) tester is thus made of a big loop containing two simple pieces of logic:
- a button-basher, that will input many combinations pseudo-randomly
- a result checker, that will check for incorrect behavior
The “button-basher” part can vary from a simple random generator to various mutators and optimizers.
More advanced button-bashers will start from so called “Fuzz vectors”, which can be pure random or pre-compiled lists of interesting inputs (for instance like OWASPS’s [4]).
They will then mutate those vectors using various techniques and try to for instance optimize code coverage, like american fuzzy lop [5] does (which is used in SQLite test suite [6] f.i, and many others).
With enough sophistication, you can even pull jpegs out of thin air [7], never underestimate what an army of Monkeys can do! Given enough time they can literally tear apart anything.
The result checker can also go from simple (checking lack of crashes), to various levels of sophistication. For instance when fuzz testing a database, you can start a transaction, button-bash, then rollback, and check that you got neither crashes nor database corruption.
At that point, all you have to do is let the monkey run [8]. In more stable programs this will be as fun as watching paint dry, but in less stable programs, this can be as fun as the previous link.
Advantages / Limitations
The main advantage of Fuzz testing is that while writing a fuzz tester is usually a relatively simple affair, the amount of invalid inputs it can test is not only super-human (it can test much faster than humans), it is even super-superman (it can test cases that superman would not even think about).
A common limitation is that fully generic monkeys are not very practical: they have long run times. So it is usually necessary to inject some smarts in the button-bashing. The typical exemple would be the case of an input that includes a CRC: if you just button-bash, you will not test much beyond the CRC check. To test more, you have to make the button-bashing aware of the CRC, which means a custom monkey button-basher.
The drawback is that outside edge cases, Fuzz testing can mostly test for incorrectness, not correctness. So you still have to write those unit tests. Also it will not be fully deterministic in the long run, so when Fuzz testing finds an input that triggers a bug, it is a good practice to transform that input into a unit test, to ensure the lack of regression on that particular bug.
TBCEditor Monkey
In the case of TBCEditor, you can have a look at one of the latest versions (at the time I am writing this) in Issue 384 [9].
The TBCEditor monkey started as a simple button-basher looking for access violations, then it started stressing the undo function (bash, then undo, and check that the buffer is back to what it was), and now one that stresses the undo, then the redo.
As Lasse is fixing more and more arcane bugs, I guess the next step will soon be to give some code-coverage awareness to the monkey…
…a psychic and sadistic monkey if you will, which knows when it is inflicting more pain, so it can try to inflict more….