blob: f8bd75f403fd8556ca16d7e7269e3da3e2b2cf16 (
plain) (
blame)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
|
tests [](https://travis-ci.org/ethereum/tests)
=====
Common tests for all clients to test against. See the documentation http://ethereum-tests.readthedocs.io/
Contents of this repository
---------------------------
Do not change test files in folders:
* StateTests
* BlockchainTests
* TransactionTests
* VMTests
It is being created by the testFillers which could be found at https://github.com/ethereum/cpp-ethereum/tree/develop/test/tools/jsontests at src folder. The filler specification and wiki are in development so please ask on gitter channel for more details.
If you want to modify a test filler or add a new test please contact @winsvega at https://gitter.im/ethereum/cpp-ethereum
All files should be of the form:
```
{
"test1name":
{
"test1property1": ...,
"test1property2": ...,
...
},
"test2name":
{
"test2property1": ...,
"test2property2": ...,
...
}
}
```
Arrays are allowed, but don't use them for sets of properties - only use them for data that is clearly a continuous contiguous sequence of values.
Test Set Sanitation
-------------------
### Checkers
Several basic checks against the test-set are performed to enusre that they have been filled and are formatted correctly.
Currently, there are three types of checks that we can perform:
- `make TEST_PREFIX.format`: check that the JSON is formatted correctly.
- `make TEST_PREFIX.valid`: check that the JSON files are valid against the JSON schemas in `./JSONSchema`.
- `make TEST_PREFIX.filled`: check that the JSON tests are filled with the correct source hashes against the fillers.
The constant `TEST_PREFIX` is a path prefix to the test-set you're interested in performing the checks on.
For instance:
- `make ./src/VMTestsFiller/vmArithmeticTest.format` will check that all JSON files in `./src/VMTestsFiller/vmArithmeticTest` are formatted correctly.
- `make ./src.valid` will check that all the JSON files in `./src` are valid against the JSON schemas in `./JSONSchema`.
- `make ./BlockchainTests.filled` will check that the source hashes in the JSON tests in `./BlockchainTests` are the same as the hashes of the fillers in `./src/BlockchainTestsFiller`.
These checks are all performed by the file `./test.py`, which can be invoked on individual files as well.
Run `./test.py` with no arguments for help.
### Sanitizers
The above checkers are packaged together into sanitizers for each test-suite, marking which testsuites are passing which testers.
See the `TODO`s in the `Makefile` to see which checkers are enabled for which test-suites.
- `make sani`: will run all passing sanitizers on all passing testsuites.
- `make sani-TESTNAME`: will run just the passing sanitizers for the given testsuite.
`TESTNAME` can be one of:
- `vm`: VMTests and VMTestsFiller
- `gs`: GeneralStateTests and GeneralStateTestsFiller
- `bc`: BlockchainTests and BlockchainTestsFiller
- `tx`: TransactionTests and TransactionTestsFiller
### Runners/Fillers
The tests can also be run/filled with the same `Makefile`.
- `make run`: will use `testeth` to all the test-suites.
- `make fill`: will use `testeth` to fill all the test-suites.
- `make TEST_PREFIX.run`: runs a single testsuite.
- `make TEST_PREFIX.fill`: fills and formats a single testsuite.
FAQ
---
### I want to test my client, which version shall I use?
[The develop branch in ethereum/tests](https://github.com/ethereum/tests/tree/develop) is the version to use.
### How can I add a new test case?
[testeth guide to generate test cases](https://github.com/ethereum/testeth/blob/develop/doc/generating_tests.rst)
|
