Testing Guide#
Testing is a core part of Earth2Studio development and we have strict requirements for unit testing any new feature. Presently we expect new functionality to have over 90% coverage within reason. The core tool used for testing in Earth2Studio is PyTest and all unit tests can be found inside the test folder. Ensure that developer dependencies are install as outline in the developer overview section.
PyTest#
Earth2Studio has several configurations for testing listed below. Many unit tests, particularly ones for data sources, have timeout limits and are allowed to fail using the xfail decorator. It is normal for some tests to fail, particularly for machines with slower internet connections. Users should always look at the test summary to under to understand which tests produced an error but did not fail.
Quick Unit Test Suite#
This will run the quickest test suite which skips many tests that are longer running.
pytest test/
Standard Unit Test Suite#
To run the full unit test suite, add the --slow flag.
Compared to the quick test suite, this one will run much longer
pytest --slow test/
Complete Unit Test Suite#
The complete unit tests ran by the CI pipeline is ran with the --ci-cache.
This option enables tests that load and test full model archiectures with their model
package which can be quite expensive.
These tests will look for model files inside a specific folder unique to the CI pipeline
on Nvidia systems or whatever is set in the EARTH2STUDIO_CACHE environment variable.
See configuration section for details on this environment
variable.
pytest --slow --ci-cache test/
Warning
The full model cache is rather large (order of 10Gb) which will be downloaded running this command. The CI pipeline will already have all model files downloaded on the runner machine to skip this step. Additional information on this can be found below.
CI Model Package Cache#
In the CI pipeline a NFS mounted folder is used as a pre-cached store of all
pre-trained model checkpoint files (automodels) so each pipeline does not need to rerun
the download.
The cache folder is managed internally by the developers.
This should reflect a local cache if a user was to fetch and load all possible models.
The point of truth for generating this cache of model checkpoints should always be
test/models/test_auto_models.py.
This test all listed auto-models and create a ./cache folder with the following
command:
pytest --model-download -s test/models/test_auto_models.py
ls -l ./cache
Coverage#
Coverage is calculated using the coverage.py package configured in the pyproject.toml. Presently the CI pipeline will fail if unit test coverage drops below 90%. Running coverage with pytest just requires the following:
# Run tests
coverage pytest --slow test/
# Compile reports
coverage combine
The CI pipeline uses the commands defined in the Makefile:
make pytest
make coverage
Contributing Tests#
We expect all contributors to also provide unit tests for their feature. Tests are always evolving based on new errors discovered and features added. Right now the best way to contribute unit tests for a new feature is to modify the tests for a feature of a similar type. E.g. for a new data source, copy and modify the tests for say GFS or ARCO. For additonal infomation about the tests, reach out with an issue and the developers can provide more information.
If you are contributing a model, be sure to make the developers aware of this to properly integrate it into the CI pipeline.