aboutsummaryrefslogtreecommitdiffstats
path: root/packages/pipeline/README.md
diff options
context:
space:
mode:
authorFabio Berger <me@fabioberger.com>2019-01-12 00:12:55 +0800
committerFabio Berger <me@fabioberger.com>2019-01-12 00:12:55 +0800
commit96ff5eef644b8c4d3f8fc4ec53270991fb51f121 (patch)
tree530f43b03d2b3aa09b5f6552d4285df92d45fed0 /packages/pipeline/README.md
parent7f5a2c972bb86df77c423fb7029e9a629ecc1ede (diff)
parent2cf57a48dd2857dd5cf2f31f4c60dd47ae4d34a5 (diff)
downloaddexon-sol-tools-96ff5eef644b8c4d3f8fc4ec53270991fb51f121.tar
dexon-sol-tools-96ff5eef644b8c4d3f8fc4ec53270991fb51f121.tar.gz
dexon-sol-tools-96ff5eef644b8c4d3f8fc4ec53270991fb51f121.tar.bz2
dexon-sol-tools-96ff5eef644b8c4d3f8fc4ec53270991fb51f121.tar.lz
dexon-sol-tools-96ff5eef644b8c4d3f8fc4ec53270991fb51f121.tar.xz
dexon-sol-tools-96ff5eef644b8c4d3f8fc4ec53270991fb51f121.tar.zst
dexon-sol-tools-96ff5eef644b8c4d3f8fc4ec53270991fb51f121.zip
merge development
Diffstat (limited to 'packages/pipeline/README.md')
-rw-r--r--packages/pipeline/README.md32
1 files changed, 25 insertions, 7 deletions
diff --git a/packages/pipeline/README.md b/packages/pipeline/README.md
index 794488cac..4fc8e0ff9 100644
--- a/packages/pipeline/README.md
+++ b/packages/pipeline/README.md
@@ -147,20 +147,38 @@ set the`ZEROEX_DATA_PIPELINE_DB_URL` environment variable to a valid
starting from that block number.
7. Run the migrations and then run your new script locally and verify it works
as expected.
+8. After all tests pass and you can run the script locally, open a new PR to
+ the monorepo. Don't merge this yet!
+9. If you added any new scripts or dependencies between scripts, you will need
+ to make changes to https://github.com/0xProject/0x-pipeline-orchestration
+ and make a separate PR there. Don't merge this yet!
+10. After your PR passes code review, ask @feuGeneA or @xianny to deploy your
+ changes to the QA environment. Check the [QA Airflow dashboard](http://airflow-qa.0x.org:8080)
+ to make sure everything works correctly in the QA environment.
+11. Merge your PR to 0x-monorepo (and
+ https://github.com/0xProject/0x-pipeline-orchestration if needed). Then ask
+ @feuGeneA or @xianny to deploy to production.
+12. Monitor the [production Airflow dashboard](http://airflow.0x.org:8080) to
+ make sure everything still works.
+13. Celebrate! :tada:
#### Additional guidelines and tips:
-* Table names should be plural and separated by underscores (e.g.,
+- Table names should be plural and separated by underscores (e.g.,
`exchange_fill_events`).
-* Any table which contains data which comes directly from a third-party source
+- Any table which contains data which comes directly from a third-party source
should be namespaced in the `raw` PostgreSQL schema.
-* Column names in the database should be separated by underscores (e.g.,
+- Column names in the database should be separated by underscores (e.g.,
`maker_asset_type`).
-* Field names in entity classes (like any other fields in TypeScript) should
+- Field names in entity classes (like any other fields in TypeScript) should
be camel-cased (e.g., `makerAssetType`).
-* All timestamps should be stored as milliseconds since the Unix Epoch.
-* Use the `BigNumber` type for TypeScript code which deals with 256-bit
+- All timestamps should be stored as milliseconds since the Unix Epoch.
+- Use the `BigNumber` type for TypeScript code which deals with 256-bit
numbers from smart contracts or for any case where we are dealing with large
floating point numbers.
-* [TypeORM documentation](http://typeorm.io/#/) is pretty robust and can be a
+- [TypeORM documentation](http://typeorm.io/#/) is pretty robust and can be a
helpful resource.
+
+* Scripts/parsers should perform minimum data transformation/normalization.
+ The idea here is to have a raw data feed that will be cleaned up and
+ synthesized in a separate step.