aboutsummaryrefslogtreecommitdiffstats
path: root/eth/downloader/queue.go
Commit message (Collapse)AuthorAgeFilesLines
* eth, les: Refactor downloader peer to use structsNick Johnson2017-06-291-5/+5
|
* eth/downloader: separate state sync from queue (#14460)Felix Lange2017-06-221-281/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * eth/downloader: separate state sync from queue Scheduling of state node downloads hogged the downloader queue lock when new requests were scheduled. This caused timeouts for other requests. With this change, state sync is fully independent of all other downloads and doesn't involve the queue at all. State sync is started and checked on in processContent. This is slightly awkward because processContent doesn't have a select loop. Instead, the queue is closed by an auxiliary goroutine when state sync fails. We tried several alternatives to this but settled on the current approach because it's the least amount of change overall. Handling of the pivot block has changed slightly: the queue previously prevented import of pivot block receipts before the state of the pivot block was available. In this commit, the receipt will be imported before the state. This causes an annoyance where the pivot block is committed as fast block head even when state downloads fail. Stay tuned for more updates in this area ;) * eth/downloader: remove cancelTimeout channel * eth/downloader: retry state requests on timeout * eth/downloader: improve comment * eth/downloader: mark peers idle when state sync is done * eth/downloader: move pivot block splitting to processContent This change also ensures that pivot block receipts aren't imported before the pivot block itself. * eth/downloader: limit state node retries * eth/downloader: improve state node error handling and retry check * eth/downloader: remove maxStateNodeRetries It fails the sync too much. * eth/downloader: remove last use of cancelCh in statesync.go Fixes TestDeliverHeadersHang*Fast and (hopefully) the weird cancellation behaviour at the end of fast sync. * eth/downloader: fix leak in runStateSync * eth/downloader: don't run processFullSyncContent in LightSync mode * eth/downloader: improve comments * eth/downloader: fix vet, megacheck * eth/downloader: remove unrequested tasks anyway * eth/downloader, trie: various polishes around duplicate items This commit explicitly tracks duplicate and unexpected state delieveries done against a trie Sync structure, also adding there to import info logs. The commit moves the db batch used to commit trie changes one level deeper so its flushed after every node insertion. This is needed to avoid a lot of duplicate retrievals caused by inconsistencies between Sync internals and database. A better approach is to track not-yet-written states in trie.Sync and flush on commit, but I'm focuing on correctness first now. The commit fixes a regression around pivot block fail count. The counter previously was reset to 1 if and only if a sync cycle progressed (inserted at least 1 entry to the database). The current code reset it already if a node was delivered, which is not stong enough, because unless it ends up written to disk, an attacker can just loop and attack ad infinitum. The commit also fixes a regression around state deliveries and timeouts. The old downloader tracked if a delivery is stale (none of the deliveries were requestedt), in which case it didn't mark the node idle and did not send further requests, since it signals a past timeout. The current code did mark it idle even on stale deliveries, which eventually caused two requests to be in flight at the same time, making the deliveries always stale and mass duplicating retrievals between multiple peers. * eth/downloader: fix state request leak This commit fixes the hang seen sometimes while doing the state sync. The cause of the hang was a rare combination of events: request state data from peer, peer drops and reconnects almost immediately. This caused a new download task to be assigned to the peer, overwriting the old one still waiting for a timeout, which in turned leaked the requests out, never to be retried. The fix is to ensure that a task assignment moves any pending one back into the retry queue. The commit also fixes a regression with peer dropping due to stalls. The current code considered a peer stalling if they timed out delivering 1 item. However, the downloader never requests only one, the minimum is 2 (attempt to fine tune estimated latency/bandwidth). The fix is simply to drop if a timeout is detected at 2 items. Apart from the above bugfixes, the commit contains some code polishes I made while debugging the hang. * core, eth, trie: support batched trie sync db writes * trie: rename SyncMemCache to syncMemBatch
* common, eth/downloader, log: support terminal log formattingPéter Szilágyi2017-02-281-9/+9
|
* eth/downloader: port over old logs from glog to log15Péter Szilágyi2017-02-271-11/+11
|
* all: blidly swap out glog to our log15, logs need reworkPéter Szilágyi2017-02-231-13/+12
|
* eth/downloader: improve deliverNodeData (#3588)Felix Lange2017-01-241-72/+52
| | | | | Commit d3b751e accidentally deleted a crucial 'return' statement, leading to a crash in case of an issue with node data. This change improves the fix in PR #3591 by removing the lock entirely.
* eth/downloader: fix mutex regression causing panics on fail (#3591)Péter Szilágyi2017-01-201-1/+2
|
* all: gofmt -w -sFelix Lange2017-01-061-2/+2
|
* trie: remove dependency on ethdbFelix Lange2017-01-061-5/+10
| | | | This removes the core/types -> leveldb dependency.
* eth/downloader: clear pending requests when switching trie rootPéter Szilágyi2016-11-011-9/+18
|
* core/state, eth/downloader, trie: reset fast-failure on progressPéter Szilágyi2016-10-311-8/+10
|
* eth, eth/downloader, eth/fetcher: delete eth/61 codeFelix Lange2016-07-221-136/+2
| | | | | The eth/61 protocol was disabled in #2776, this commit removes its message handlers and hash-chain sync logic.
* eth/downloader, trie: pull head state concurrently with chainPéter Szilágyi2016-05-271-1/+7
|
* eth/downloader: fix reviewer commentsPéter Szilágyi2016-05-201-2/+2
|
* eth/downloader: stream partial skeleton filling to processorPéter Szilágyi2016-05-171-6/+29
|
* eth/downloader: implement concurrent header downloadsPéter Szilágyi2016-05-171-2/+200
|
* all: fix go vet warningsFelix Lange2016-04-151-1/+1
|
* eth: various typosLeif Jurvetson2016-03-161-1/+1
|
* Merge pull request #2251 from karalabe/increase-downlaoder-queuePéter Szilágyi2016-02-241-1/+1
|\ | | | | eth/downloader: bump the download queue size to prevent starvation
| * eth/downloader: bump the download queue size to prevent starvationPéter Szilágyi2016-02-231-1/+1
| |
* | all: Rename crypto.Sha3{,Hash}() to crypto.Keccak256{,Hash}()Ricardo Catalinas Jiménez2016-02-221-1/+1
|/ | | | As we aren't really using the standarized SHA-3
* eth/downloader: raise pending state limit that prevented concurrencyPéter Szilágyi2016-02-081-2/+3
|
* eth/downloader: fetch data proportionally to peer capacityPéter Szilágyi2015-11-191-36/+59
|
* eth/downloader: don't hang for spurious deliveriesFelix Lange2015-11-191-97/+101
| | | | | | | | | | | | | | | | Unexpected deliveries could block indefinitely if they arrived at the right time. The fix is to ensure that the cancellation channel is always closed when the sync ends, unblocking any deliveries. Also remove the atomic check for whether a sync is currently running because it doesn't help and can be misleading. Cancelling always seems to break the tests though. The downloader spawned d.process whenever new data arrived, making it somewhat hard to track when block processing was actually done. Fix this by running d.process in a dedicated goroutine that is tied to the lifecycle of the sync. d.process gets notified of new work by the queue instead of being invoked all the time. This removes a ton of weird workaround code, including a hairy use of atomic CAS.
* eth/downloader: fix dysfunctional ignore list hidden by generic setPéter Szilágyi2015-11-041-6/+6
|
* core, eth, trie: fix data races and merge/review issuesPéter Szilágyi2015-10-211-58/+120
|
* eth: clean out light node notions from ethPéter Szilágyi2015-10-191-4/+6
|
* eth/downloader: concurrent receipt and state processingPéter Szilágyi2015-10-191-45/+115
|
* core, eth, trie: direct state trie synchronizationPéter Szilágyi2015-10-191-55/+216
|
* eth/downloader: add fast and light sync strategiesPéter Szilágyi2015-10-191-190/+344
|
* eth/downloader: header-chain order and ancestry checkPéter Szilágyi2015-09-151-2/+15
|
* eth: update metrics collection to handle eth/62 algosPéter Szilágyi2015-08-251-0/+12
|
* eth: port the synchronisation algo to eth/62Péter Szilágyi2015-08-251-32/+207
|
* all: fix license headers one more timeFelix Lange2015-07-241-1/+1
| | | | I forgot to update one instance of "go-ethereum" in commit 3f047be5a.
* all: update license headers to distiguish GPL/LGPLFelix Lange2015-07-231-4/+4
| | | | | All code outside of cmd/ is licensed as LGPL. The headers now reflect this by calling the whole work "the go-ethereum library".
* all: update license informationFelix Lange2015-07-071-0/+16
|
* eth, eth/downloader: transition to eth 61Péter Szilágyi2015-07-011-12/+16
|
* eth, eth/downloader: move peer removal into downloaderPéter Szilágyi2015-06-151-1/+1
|
* eth/downloader: cap the hash ban set, add test for itPéter Szilágyi2015-06-081-1/+1
|
* eth/downloader: fix another rebase errorPéter Szilágyi2015-06-081-2/+1
|
* eth/downloader: preallocate the block cachePéter Szilágyi2015-06-081-16/+9
|
* eth/downloader: differentiate stale and nonexistent deliveriesPéter Szilágyi2015-06-051-3/+12
|
* eth/downloader: fix #1178, don't request blocks beyond the cache boundsPéter Szilágyi2015-06-041-1/+1
|
* eth/downloader: add a basic block download congestion controlPéter Szilágyi2015-06-031-2/+8
|
* eth, eth/downloader: surface downloaded block origin, drop on errorPéter Szilágyi2015-05-261-9/+11
|
* eth, eth/downloader: remove duplicate consts, bump hash fetch to 2KPéter Szilágyi2015-05-211-1/+1
|
* eth, eth/downloader: detect and handle madeup hash attacksPéter Szilágyi2015-05-151-4/+6
|
* eth/downloader: circumvent hash reordering attacksPéter Szilágyi2015-05-151-6/+5
|
* eth, eth/downloader: prevent hash repeater attackPéter Szilágyi2015-05-151-9/+11
|
* eth, eth/downloader: remove parent verification from the downlaoderPéter Szilágyi2015-05-141-8/+2
|
* eth/downloader: fix #910, thread safe peers & polishesPéter Szilágyi2015-05-111-0/+9
|
* eth, eth/downloader: make synchronize thread safePéter Szilágyi2015-05-081-10/+0
|
* eth/downloader: fix expiration not running while fetchingPéter Szilágyi2015-05-071-12/+14
|
* eth/downloader: prioritize block fetch based on chain position, cap memory usePéter Szilágyi2015-05-071-134/+282
|
* eth/downloader: put back hashes on block overflow errorobscuren2015-05-031-19/+14
|
* eth, eth/downloader: Moved block processing & graceful shutdownobscuren2015-05-011-4/+54
| | | | | | | The downloader is no longer responsible for processing blocks. The eth-protocol handler now takes care of this instead. Added graceful shutdown during block processing. Closes #846
* eth/downloader: removed peer td management and best peer selectionobscuren2015-04-241-2/+1
|
* downloader: moved chunk ignoring. Fixes issue with catching upobscuren2015-04-191-7/+15
|
* downloader: free up peers from work when the downloader resetsobscuren2015-04-191-4/+9
|
* downloader: improved downloading and synchronisationobscuren2015-04-191-0/+3
| | | | | | * Downloader's peers keeps track of peer's previously requested hashes so that we don't have to re-request * Changed `AddBlock` to be fully synchronous
* downloader: reset the queue if a peer response with an empty hash setobscuren2015-04-181-0/+11
|
* downloader: added missing blocks catchup functionalityobscuren2015-04-131-6/+38
| | | | | When a parent is missing in the block list an attempt should be made to fetch the missing parent and grandparents.
* downloader: renamed chunks to queueobscuren2015-04-121-0/+98