aboutsummaryrefslogtreecommitdiffstats
path: root/packages/pipeline/README.md
diff options
context:
space:
mode:
authorHsuan Lee <hsuan@cobinhood.com>2019-01-19 18:42:04 +0800
committerHsuan Lee <hsuan@cobinhood.com>2019-01-19 18:42:04 +0800
commit7ae38906926dc09bc10670c361af0d2bf0050426 (patch)
tree5fb10ae366b987db09e4ddb4bc3ba0f75404ad08 /packages/pipeline/README.md
parentb5fd3c72a08aaa6957917d74c333387a16edf66b (diff)
downloaddexon-sol-tools-7ae38906926dc09bc10670c361af0d2bf0050426.tar
dexon-sol-tools-7ae38906926dc09bc10670c361af0d2bf0050426.tar.gz
dexon-sol-tools-7ae38906926dc09bc10670c361af0d2bf0050426.tar.bz2
dexon-sol-tools-7ae38906926dc09bc10670c361af0d2bf0050426.tar.lz
dexon-sol-tools-7ae38906926dc09bc10670c361af0d2bf0050426.tar.xz
dexon-sol-tools-7ae38906926dc09bc10670c361af0d2bf0050426.tar.zst
dexon-sol-tools-7ae38906926dc09bc10670c361af0d2bf0050426.zip
Update dependency packages
Diffstat (limited to 'packages/pipeline/README.md')
-rw-r--r--packages/pipeline/README.md184
1 files changed, 0 insertions, 184 deletions
diff --git a/packages/pipeline/README.md b/packages/pipeline/README.md
deleted file mode 100644
index 4fc8e0ff9..000000000
--- a/packages/pipeline/README.md
+++ /dev/null
@@ -1,184 +0,0 @@
-## @0xproject/pipeline
-
-This repository contains scripts used for scraping data from the Ethereum blockchain into SQL tables for analysis by the 0x team.
-
-## Contributing
-
-We strongly recommend that the community help us make improvements and determine the future direction of the protocol. To report bugs within this package, please create an issue in this repository.
-
-Please read our [contribution guidelines](../../CONTRIBUTING.md) before getting started.
-
-### Install dependencies:
-
-```bash
-yarn install
-```
-
-### Build
-
-```bash
-yarn build
-```
-
-### Clean
-
-```bash
-yarn clean
-```
-
-### Lint
-
-```bash
-yarn lint
-```
-
-### Migrations
-
-Create a new migration: `yarn migrate:create --name MigrationNameInCamelCase`
-Run migrations: `yarn migrate:run`
-Revert the most recent migration (CAUTION: may result in data loss!): `yarn migrate:revert`
-
-## Testing
-
-There are several test scripts in **package.json**. You can run all the tests
-with `yarn test:all` or run certain tests seprately by following the
-instructions below. Some tests may not work out of the box on certain platforms
-or operating systems (see the "Database tests" section below).
-
-### Unit tests
-
-The unit tests can be run with `yarn test`. These tests don't depend on any
-services or databases and will run in any environment that can run Node.
-
-### Database tests
-
-Database integration tests can be run with `yarn test:db`. These tests will
-attempt to automatically spin up a Postgres database via Docker. If this doesn't
-work you have two other options:
-
-1. Set the `DOCKER_SOCKET` environment variable to a valid socket path to use
- for communicating with Docker.
-2. Start Postgres manually and set the `ZEROEX_DATA_PIPELINE_TEST_DB_URL`
- environment variable. If this is set, the tests will use your existing
- Postgres database instead of trying to create one with Docker.
-
-## Running locally
-
-`pipeline` requires access to a PostgreSQL database. The easiest way to start
-Postgres is via Docker. Depending on your platform, you may need to prepend
-`sudo` to the following command:
-
-```
-docker run --rm -d -p 5432:5432 --name pipeline_postgres postgres:11-alpine
-```
-
-This will start a Postgres server with the default username and database name
-(`postgres` and `postgres`). You should set the environment variable as follows:
-
-```
-export ZEROEX_DATA_PIPELINE_DB_URL=postgresql://postgres@localhost/postgres
-```
-
-First thing you will need to do is run the migrations:
-
-```
-yarn migrate:run
-```
-
-Now you can run scripts locally:
-
-```
-node packages/pipeline/lib/src/scripts/pull_radar_relay_orders.js
-```
-
-To stop the Postgres server (you may need to add `sudo`):
-
-```
-docker stop pipeline_postgres
-```
-
-This will remove all data from the database.
-
-If you prefer, you can also install Postgres with e.g.,
-[Homebrew](https://wiki.postgresql.org/wiki/Homebrew) or
-[Postgress.app](https://postgresapp.com/). Keep in mind that you will need to
-set the`ZEROEX_DATA_PIPELINE_DB_URL` environment variable to a valid
-[PostgreSQL connection url](https://stackoverflow.com/questions/3582552/postgresql-connection-url)
-
-## Directory structure
-
-```
-.
-├── lib: Code generated by the TypeScript compiler. Don't edit this directly.
-├── migrations: Code for creating and updating database schemas.
-├── node_modules:
-├── src: All TypeScript source code.
-│   ├── data_sources: Code responsible for getting raw data, typically from a third-party source.
-│   ├── entities: TypeORM entities which closely mirror our database schemas. Some other ORMs call these "models".
-│   ├── parsers: Code for converting raw data into entities.
-│   ├── scripts: Executable scripts which put all the pieces together.
-│   └── utils: Various utils used across packages/files.
-├── test: All tests go here and are organized in the same way as the folder/file that they test.
-```
-
-## Adding new data to the pipeline
-
-1. Create an entity in the _entities_ directory. Entities directly mirror our
- database schemas. We follow the practice of having "dumb" entities, so
- entity classes should typically not have any methods.
-2. Create a migration using the `yarn migrate:create` command. Create/update
- tables as needed. Remember to fill in both the `up` and `down` methods. Try
- to avoid data loss as much as possible in your migrations.
-3. Add basic tests for your entity and migrations to the **test/entities/**
- directory.
-4. Create a class or function in the **data_sources/** directory for getting
- raw data. This code should abstract away pagination and rate-limiting as
- much as possible.
-5. Create a class or function in the **parsers/** directory for converting the
- raw data into an entity. Also add tests in the **tests/** directory to test
- the parser.
-6. Create an executable script in the **scripts/** directory for putting
- everything together. Your script can accept environment variables for things
- like API keys. It should pull the data, parse it, and save it to the
- database. Scripts should be idempotent and atomic (when possible). What this
- means is that your script may be responsible for determining _which_ data
- needs to be updated. For example, you may need to query the database to find
- the most recent block number that we have already pulled, then pull new data
- starting from that block number.
-7. Run the migrations and then run your new script locally and verify it works
- as expected.
-8. After all tests pass and you can run the script locally, open a new PR to
- the monorepo. Don't merge this yet!
-9. If you added any new scripts or dependencies between scripts, you will need
- to make changes to https://github.com/0xProject/0x-pipeline-orchestration
- and make a separate PR there. Don't merge this yet!
-10. After your PR passes code review, ask @feuGeneA or @xianny to deploy your
- changes to the QA environment. Check the [QA Airflow dashboard](http://airflow-qa.0x.org:8080)
- to make sure everything works correctly in the QA environment.
-11. Merge your PR to 0x-monorepo (and
- https://github.com/0xProject/0x-pipeline-orchestration if needed). Then ask
- @feuGeneA or @xianny to deploy to production.
-12. Monitor the [production Airflow dashboard](http://airflow.0x.org:8080) to
- make sure everything still works.
-13. Celebrate! :tada:
-
-#### Additional guidelines and tips:
-
-- Table names should be plural and separated by underscores (e.g.,
- `exchange_fill_events`).
-- Any table which contains data which comes directly from a third-party source
- should be namespaced in the `raw` PostgreSQL schema.
-- Column names in the database should be separated by underscores (e.g.,
- `maker_asset_type`).
-- Field names in entity classes (like any other fields in TypeScript) should
- be camel-cased (e.g., `makerAssetType`).
-- All timestamps should be stored as milliseconds since the Unix Epoch.
-- Use the `BigNumber` type for TypeScript code which deals with 256-bit
- numbers from smart contracts or for any case where we are dealing with large
- floating point numbers.
-- [TypeORM documentation](http://typeorm.io/#/) is pretty robust and can be a
- helpful resource.
-
-* Scripts/parsers should perform minimum data transformation/normalization.
- The idea here is to have a raw data feed that will be cleaned up and
- synthesized in a separate step.