Test Matrix
Boost Member for 10 years
Test Matrix
-
https://www.boost.org/doc/contributor-guide/testing/boost-test-matrix.html#_introduction">Introduction
-
https://www.boost.org/doc/contributor-guide/testing/boost-test-matrix.html#_regression_dashboard">Regression Dashboard
-
https://www.boost.org/doc/contributor-guide/testing/boost-test-matrix.html#_handling_test_failures">Handling Test Failures
-
https://www.boost.org/doc/contributor-guide/testing/boost-test-matrix.html#_boost_build">Boost Build
-
https://www.boost.org/doc/contributor-guide/testing/boost-test-matrix.html#_test_process_feedback">Test Process Feedback
-
https://www.boost.org/doc/contributor-guide/testing/boost-test-matrix.html#_see_also">See Also
Introduction
The Boost Test Matrix is an automated testing system that runs tests on Boost libraries across a wide range of platforms, compiler versions, and configurations. Its primary purpose is to ensure that the libraries work correctly under various conditions and to identify any compatibility issues.
The Test Matrix includes tests run on different operating systems (Windows, Linux, macOS) and with various compilers (such as GCC, Clang, MSVC). This diversity helps in catching issues that might only appear in specific environments.
For information on running regression tests locally, refer to https://www.boost.org/doc/contributor-guide/testing/regression-tests.html">Local Regression Tests.
Regression Dashboard
The results of library tests are published on the Boost Regression Testing Dashboard:
Version | Results | Issues |
|---|---|---|
Develop branch | https://regression.boost.org/develop/developer/summary.html">Summary | https://regression.boost.org/develop/developer/issues.html">Unresolved Issues |
Master branch | https://regression.boost.org/master/developer/summary.html">Summary | https://regression.boost.org/master/developer/issues.html">Unresolved Issues |
This dashboard is publicly accessible and provides detailed information about the test results for most libraries.
Handling Test Failures
If your contribution causes test failures, it’s expected that you take responsibility for fixing them. This might involve making code adjustments or collaborating with others if the issue is complex, or affects multiple parts of Boost.
On the dashboard, you will find a matrix of test results. The columns represent different test runners (which correspond to different platforms, compiler versions, etc.), and the rows represent individual libraries.
When you click on the Dashboard URL, you will see the test Summary.
https://www.boost.org/doc/contributor-guide/_images/test-matrix-summary.png">
The table is large, and you will need to scroll up and down and left and right to view all relevant results. Note that some entries are underlined (even OK entries) which indicates they are links to further information.
https://www.boost.org/doc/contributor-guide/_images/test-matrix-sample.png">
Dashboard Legend
The dashboard uses different colors or labels to indicate the status of the tests (such as 'pass', 'fail', or 'unresolved'). Understanding these labels is key to interpreting the test results correctly. Note that some tests are expected to fail.
https://www.boost.org/doc/contributor-guide/_images/test-matrix-legend.png">
By clicking on specific cells in the matrix, you can view detailed results for a particular library and test runner. This includes information about which tests passed, which failed, and often detailed logs of the test runs.
https://www.boost.org/doc/contributor-guide/_images/test-matrix-sample-2.png">
Example
For example, click on the https://www.boost.org/libs/asio">Boost.Asio entry in the left-hand column, and you will bring up the test matrix for the Asio/Master library.