This removes --wconsole, --wnoconsole, and --wnoredirect. Instead these are now handled by --no-log-to-file and --log-to-file.
If logging to file is enabled, then output will be written to the log file. If logging to file is not enabled, then output is written to the terminal. On Windows, a terminal will be created for the output to be written to if wesnoth is not launched from a terminal.
The fix in 9535ed9 didn't work, the script still crashed on that line, the
logic for working out how many tests in the batch had passed wasn't correct.
This commit drops that logic, and drops the dependent feature of trying to
retry the next test in the batch, and simply says the entire batch failed.
The new logic for filtering the output of a failed batch is much simpler - the
script doesn't filter it and we assume the developer is just going to ignore
any line that starts "PASS TEST". Only tests that are expected to pass get
batched, so the output is going to look something like the following:
PASS TEST: test_return
PASS TEST: test_assert
PASS TEST: test_return_2
PASS TEST: test_assert_2
Assertion failed: assert false
FAIL TEST: test_assert_fail
Failure, Wesnoth returned 1 FAIL but we expected 0 PASS
A set of constants is added defining how many -v arguments are needed to
enable a given set of debugging output. Expectation is that level 0 is normal,
and level 1 is for developers who like long-running build processes to print
something to show that they're not stuck.
Previously, the runner relied on the name being part of the output log from the Wesnoth process.
This works in most cases because Wesnoth prints a line with the test's name before exiting.
However, if the test timed out, that line would typically be missing, so this makes the runner insert it into the failure message as well to make sure it's present.
This allows passing an explicit path on Mac (instead of using the automatic -p xcode option), since the binary is not named "wesnoth" on that platform.
- Avoids creating tons of external files
- Suppresses the startup banner to reduce extraneous messages
- Makes the system work on Windows
- Fixes an issue of batched tests being misnamed if one of them is an error
This adds an additional `test_result` attribute to [endlevel], intended for use with the automated unit tests. This allows for the unit tests to differentiate a pass/fail result separately from scenario victory or defeat, which allows for more accurately determining the outcome of a test as well as addresses the potential, for example, for a scenario to be expect to pass because of the {SUCCEED} macro but instead passes because the scenario ended as a victory through some other method.
Additional unit tests which were the original motivation for this change are also added as part of this. They test, as much as possible, that events are executed at all, and are then also executed in the expected order.
This is part of working out whether a subset of the "fail" tests could be run
in one Wesnoth instance. To run a test that returns TEST_FAIL_BROKE_STRICT with
any other test would require a mechanism to reset lg::broke_strict()'s flag.
All tests that fail with an {ASSERT} will also set the lg::broke_strict() flag,
the tests with the new status are only those that would pass without the strict
flag.
In the SUF tests, change a test from fail-on-success to breaks-strict, rename
the formula tests and add some comments. The rename is because "fai" is
"Formula AI", an obsolete name for WFL.
This commit:
* Adds the new `bt` (batch timeout) option, to limit the maximum time batched unit tests can take. Otherwise, if for whatever reason they take longer than 10 minutes, travis will error the job due to not receiving any output for too long.
* Writes the test output to files rather than trying to get it from the stderr, and then output's the log from the failed test. Currently if a test unexpectedly times out, there is nothing printed indicating the error or which test encountered the problem (in case of batched tests).
The verbosity settings are a bit different, but other than that it
should be a drop-in replacement for the old shell script.
Wesnoth's --validcache option isn't used any more, which is part of
the cleanup for #4587 (it didn't speed anything up, because all of
the calculation for the checksum was still done).
The ability isn't needed in master because none of the unit tests call
deprecated functions here, but it may be useful in the future or make
cherry-picking of future changes easier.
(cherry-picked from commit 74d8cfa98e)
The check to see whether the timeout command is capable of sending
SIGKILL was broken. This fixes the check, which in turn will ensure
that tests are always terminated. This will resolve travis deciding
that the entire build has timed out.
This explains (in the log) why a WML conditional has failed (or succeeded). Currently it is fairly basic, looping through and/or/not and printing out the specific conditional tag that failed as well as, for [variable], the current content of the variable.
It's used in the WML unit tests system, but could also be useful for debugging.
This commit also fixes an issue with passing multiple -a arguments to run_wml_tests.
- If timeout is missing, it disables timeout and skips any tests that expect it
- Pass -t0 to get the above behaviour even if timeout is present
- Detect if timeout doesn't support --kill-after
- Mac users: Pass -pxcode to automatically find the Wesnoth binary in XCode's DerivedProducts folder
- In non-verbose mode, give progress report by printing a dot before each test
This only happens in non-verbose mode because this method of reporting progress makes most sense
if nothing else is printed.
This was buggy and based on deprecated SDL features that will not
continue to be available. aquileia wrote us a very nice windows
port of the run_wml_tests bash script, so for windows mac and
linux we should now be able to run the unit tests as desired.
Maybe someday built-in timeouts can be supported using boost
process library, if that ever materializes.
