diff options
Diffstat (limited to 'README.md')
| -rw-r--r-- | README.md | 45 |
1 files changed, 45 insertions, 0 deletions
@@ -37,6 +37,51 @@ All files should be of the form: Arrays are allowed, but don't use them for sets of properties - only use them for data that is clearly a continuous contiguous sequence of values. +Test Set Sanitation +------------------- + +### Checkers + +Several basic checks against the test-set are performed to enusre that they have been filled and are formatted correctly. +Currently, there are three types of checks that we can perform: + +- `make TEST_PREFIX.format`: check that the JSON is formatted correctly. +- `make TEST_PREFIX.valid`: check that the JSON files are valid against the JSON schemas in `./JSONSchema`. +- `make TEST_PREFIX.filled`: check that the JSON tests are filled with the correct source hashes against the fillers. + +The constant `TEST_PREFIX` is a path prefix to the test-set you're interested in performing the checks on. +For instance: + +- `make ./src/VMTestsFiller/vmArithmeticTest.format` will check that all JSON files in `./src/VMTestsFiller/vmArithmeticTest` are formatted correctly. +- `make ./src.valid` will check that all the JSON files in `./src` are valid against the JSON schemas in `./JSONSchema`. +- `make ./BlockchainTests.filled` will check that the source hashes in the JSON tests in `./BlockchainTests` are the same as the hashes of the fillers in `./src/BlockchainTestsFiller`. + +These checks are all performed by the file `./test.py`, which can be invoked on individual files as well. +Run `./test.py` with no arguments for help. + +### Sanitizers + +The above checkers are packaged together into sanitizers for each test-suite, marking which testsuites are passing which testers. +See the `TODO`s in the `Makefile` to see which checkers are enabled for which test-suites. + +- `make sani`: will run all passing sanitizers on all passing testsuites. +- `make sani-TESTNAME`: will run just the passing sanitizers for the given testsuite. + `TESTNAME` can be one of: + + - `vm`: VMTests and VMTestsFiller + - `gs`: GeneralStateTests and GeneralStateTestsFiller + - `bc`: BlockchainTests and BlockchainTestsFiller + - `tx`: TransactionTests and TransactionTestsFiller + +### Runners/Fillers + +The tests can also be run/filled with the same `Makefile`. + +- `make run`: will use `testeth` to all the test-suites. +- `make fill`: will use `testeth` to fill all the test-suites. +- `make TEST_PREFIX.run`: runs a single testsuite. +- `make TEST_PREFIX.fill`: fills and formats a single testsuite. + FAQ --- |