Previously, the runner relied on the name being part of the output log from the Wesnoth process.
This works in most cases because Wesnoth prints a line with the test's name before exiting.
However, if the test timed out, that line would typically be missing, so this makes the runner insert it into the failure message as well to make sure it's present.
This allows passing an explicit path on Mac (instead of using the automatic -p xcode option), since the binary is not named "wesnoth" on that platform.
- Avoids creating tons of external files
- Suppresses the startup banner to reduce extraneous messages
- Makes the system work on Windows
- Fixes an issue of batched tests being misnamed if one of them is an error
This adds an additional `test_result` attribute to [endlevel], intended for use with the automated unit tests. This allows for the unit tests to differentiate a pass/fail result separately from scenario victory or defeat, which allows for more accurately determining the outcome of a test as well as addresses the potential, for example, for a scenario to be expect to pass because of the {SUCCEED} macro but instead passes because the scenario ended as a victory through some other method.
Additional unit tests which were the original motivation for this change are also added as part of this. They test, as much as possible, that events are executed at all, and are then also executed in the expected order.
This is part of working out whether a subset of the "fail" tests could be run
in one Wesnoth instance. To run a test that returns TEST_FAIL_BROKE_STRICT with
any other test would require a mechanism to reset lg::broke_strict()'s flag.
All tests that fail with an {ASSERT} will also set the lg::broke_strict() flag,
the tests with the new status are only those that would pass without the strict
flag.
In the SUF tests, change a test from fail-on-success to breaks-strict, rename
the formula tests and add some comments. The rename is because "fai" is
"Formula AI", an obsolete name for WFL.
This commit:
* Adds the new `bt` (batch timeout) option, to limit the maximum time batched unit tests can take. Otherwise, if for whatever reason they take longer than 10 minutes, travis will error the job due to not receiving any output for too long.
* Writes the test output to files rather than trying to get it from the stderr, and then output's the log from the failed test. Currently if a test unexpectedly times out, there is nothing printed indicating the error or which test encountered the problem (in case of batched tests).
The verbosity settings are a bit different, but other than that it
should be a drop-in replacement for the old shell script.
Wesnoth's --validcache option isn't used any more, which is part of
the cleanup for #4587 (it didn't speed anything up, because all of
the calculation for the checksum was still done).
The ability isn't needed in master because none of the unit tests call
deprecated functions here, but it may be useful in the future or make
cherry-picking of future changes easier.
(cherry-picked from commit 74d8cfa98e0f61ee22a536fc06c2019c9c89aede)
Though given 00d87f8fe4078ffd84fcdc0d73fcc2104efef1e7 I would say this is better than nothing.
(cherry-picked from commit 33ad6be326c234b173d680ce34589464f88f8418)
The check to see whether the timeout command is capable of sending
SIGKILL was broken. This fixes the check, which in turn will ensure
that tests are always terminated. This will resolve travis deciding
that the entire build has timed out.
This explains (in the log) why a WML conditional has failed (or succeeded). Currently it is fairly basic, looping through and/or/not and printing out the specific conditional tag that failed as well as, for [variable], the current content of the variable.
It's used in the WML unit tests system, but could also be useful for debugging.
This commit also fixes an issue with passing multiple -a arguments to run_wml_tests.
- If timeout is missing, it disables timeout and skips any tests that expect it
- Pass -t0 to get the above behaviour even if timeout is present
- Detect if timeout doesn't support --kill-after
- Mac users: Pass -pxcode to automatically find the Wesnoth binary in XCode's DerivedProducts folder
- In non-verbose mode, give progress report by printing a dot before each test
This only happens in non-verbose mode because this method of reporting progress makes most sense
if nothing else is printed.
This was buggy and based on deprecated SDL features that will not
continue to be available. aquileia wrote us a very nice windows
port of the run_wml_tests bash script, so for windows mac and
linux we should now be able to run the unit tests as desired.
Maybe someday built-in timeouts can be supported using boost
process library, if that ever materializes.
- travis runs tests not in verbose mode,
*but* if there is a failure,
then it runs in very verbose mode, with strict mode off, to get
errors reported in the travis log.
- run_wml_tests points out that if you don't run in strict mode,
some tests may not fail as expected, so the count at the end
may differ
