Though given 00d87f8fe4078ffd84fcdc0d73fcc2104efef1e7 I would say this is better than nothing.
(cherry-picked from commit 33ad6be326c234b173d680ce34589464f88f8418)
We keep getting segfaults that don't reproduce the second time it
is run, so maybe this will catch them.
It also moves the test execution code out into a script.