摘要:當前代碼是以太坊,如果版本不同,代碼上可能存在差異。非產生區塊節點的策略圖,如圖,黃色節點將區塊傳播給青色節點至此,可以看出以太坊采用以石擊水的方式,像水紋一樣,層層擴散新產生的區塊。
前言
這篇文章從區塊傳播策略入手,介紹新區塊是如何傳播到遠端節點,以及新區塊加入到遠端節點本地鏈的過程,同時會介紹fetcher模塊,fetcher的功能是處理Peer通知的區塊信息。在介紹過程中,還會涉及到p2p,eth等模塊,不會專門介紹,而是專注區塊的傳播和加入區塊鏈的過程。
當前代碼是以太坊Release 1.8,如果版本不同,代碼上可能存在差異。
總體過程和傳播策略本節從宏觀角度介紹,節點產生區塊后,為了傳播給遠端節點做了啥,遠端節點收到區塊后又做了什么,每個節點都連接了很多Peer,它傳播的策略是什么樣的?
總體流程和策略可以總結為,傳播給遠端Peer節點,Peer驗證區塊無誤后,加入到本地區塊鏈,繼續傳播新區塊信息。具體過程如下。
先看總體過程。產生區塊后,miner模塊會發布一個事件NewMinedBlockEvent,訂閱事件的協程收到事件后,就會把新區塊的消息,廣播給它所有的peer,peer收到消息后,會交給自己的fetcher模塊處理,fetcher進行基本的驗證后,區塊沒問題,發現這個區塊就是本地鏈需要的下一個區塊,則交給blockChain進一步進行完整的驗證,這個過程會執行區塊所有的交易,無誤后把區塊加入到本地鏈,寫入數據庫,這個過程就是下面的流程圖,圖1。
總體流程圖,能看到有個分叉,是因為節點傳播新區塊是有策略的。它的傳播策略為:
假如節點連接了N個Peer,它只向Peer列表的sqrt(N)個Peer廣播完整的區塊消息。
向所有的Peer廣播只包含區塊Hash的消息。
策略圖的效果如圖2,紅色節點將區塊傳播給黃色節點:
收到區塊Hash的節點,需要從發送給它消息的Peer那里獲取對應的完整區塊,獲取區塊后就會按照圖1的流程,加入到fetcher隊列,最終插入本地區塊鏈后,將區塊的Hash值廣播給和它相連,但還不知道這個區塊的Peer。非產生區塊節點的策略圖,如圖3,黃色節點將區塊Hash傳播給青色節點:
至此,可以看出以太坊采用以石擊水的方式,像水紋一樣,層層擴散新產生的區塊。
Fetcher模塊是干啥的fetcher模塊的功能,就是收集其他Peer通知它的區塊信息:1)完整的區塊2)區塊Hash消息。根據通知的消息,獲取完整的區塊,然后傳遞給eth模塊把區塊插入區塊鏈。
如果是完整區塊,就可以傳遞給eth插入區塊,如果只有區塊Hash,則需要從其他的Peer獲取此完整的區塊,然后再傳遞給eth插入區塊。
源碼解讀本節介紹區塊傳播和處理的細節東西,方式仍然是先用圖解釋流程,再是代碼流程。
產塊節點的傳播新區塊節點產生區塊后,廣播的流程可以表示為圖4:
發布事件
事件處理函數選擇要廣播完整的Peer,然后將區塊加入到它們的隊列
事件處理函數把區塊Hash添加到所有Peer的另外一個通知隊列
每個Peer的廣播處理函數,會遍歷它的待廣播區塊隊列和通知隊列,把數據封裝成消息,調用P2P接口發送出去
再看下代碼上的細節。
worker.wait()函數發布事件NewMinedBlockEvent。
ProtocolManager.minedBroadcastLoop()是事件處理函數。它調用了2次pm.BroadcastBlock()。
// Mined broadcast loop func (pm *ProtocolManager) minedBroadcastLoop() { // automatically stops if unsubscribe for obj := range pm.minedBlockSub.Chan() { switch ev := obj.Data.(type) { case core.NewMinedBlockEvent: pm.BroadcastBlock(ev.Block, true) // First propagate block to peers pm.BroadcastBlock(ev.Block, false) // Only then announce to the rest } } }
pm.BroadcastBlock()的入參propagate為真時,向部分Peer廣播完整的區塊,調用peer.AsyncSendNewBlock(),否則向所有Peer廣播區塊頭,調用peer.AsyncSendNewBlockHash(),這2個函數就是把數據放入隊列,此處不再放代碼。
// BroadcastBlock will either propagate a block to a subset of it"s peers, or // will only announce it"s availability (depending what"s requested). func (pm *ProtocolManager) BroadcastBlock(block *types.Block, propagate bool) { hash := block.Hash() peers := pm.peers.PeersWithoutBlock(hash) // If propagation is requested, send to a subset of the peer // 這種情況,要把區塊廣播給部分peer if propagate { // Calculate the TD of the block (it"s not imported yet, so block.Td is not valid) // 計算新的總難度 var td *big.Int if parent := pm.blockchain.GetBlock(block.ParentHash(), block.NumberU64()-1); parent != nil { td = new(big.Int).Add(block.Difficulty(), pm.blockchain.GetTd(block.ParentHash(), block.NumberU64()-1)) } else { log.Error("Propagating dangling block", "number", block.Number(), "hash", hash) return } // Send the block to a subset of our peers // 廣播區塊給部分peer transfer := peers[:int(math.Sqrt(float64(len(peers))))] for _, peer := range transfer { peer.AsyncSendNewBlock(block, td) } log.Trace("Propagated block", "hash", hash, "recipients", len(transfer), "duration", common.PrettyDuration(time.Since(block.ReceivedAt))) return } // Otherwise if the block is indeed in out own chain, announce it // 把區塊hash值廣播給所有peer if pm.blockchain.HasBlock(hash, block.NumberU64()) { for _, peer := range peers { peer.AsyncSendNewBlockHash(block) } log.Trace("Announced block", "hash", hash, "recipients", len(peers), "duration", common.PrettyDuration(time.Since(block.ReceivedAt))) } }
peer.broadcase()是每個Peer連接的廣播函數,它只廣播3種消息:交易、完整的區塊、區塊的Hash,這樣表明了節點只會主動廣播這3中類型的數據,剩余的數據同步,都是通過請求-響應的方式。
// broadcast is a write loop that multiplexes block propagations, announcements // and transaction broadcasts into the remote peer. The goal is to have an async // writer that does not lock up node internals. func (p *peer) broadcast() { for { select { // 廣播交易 case txs := <-p.queuedTxs: if err := p.SendTransactions(txs); err != nil { return } p.Log().Trace("Broadcast transactions", "count", len(txs)) // 廣播完整的新區塊 case prop := <-p.queuedProps: if err := p.SendNewBlock(prop.block, prop.td); err != nil { return } p.Log().Trace("Propagated block", "number", prop.block.Number(), "hash", prop.block.Hash(), "td", prop.td) // 廣播區塊Hash case block := <-p.queuedAnns: if err := p.SendNewBlockHashes([]common.Hash{block.Hash()}, []uint64{block.NumberU64()}); err != nil { return } p.Log().Trace("Announced block", "number", block.Number(), "hash", block.Hash()) case <-p.term: return } } }Peer節點處理新區塊
本節介紹遠端節點收到2種區塊同步消息的處理,其中NewBlockMsg的處理流程比較清晰,也簡潔。NewBlockHashesMsg消息的處理就繞了2繞,從總體流程圖1上能看出來,它需要先從給他發送消息Peer那里獲取到完整的區塊,剩下的流程和NewBlockMsg又一致了。
這部分涉及的模塊多,畫出來有種眼花繚亂的感覺,但只要抓住上面的主線,代碼看起來還是很清晰的。通過圖5先看下整體流程。
消息處理的起點是ProtocolManager.handleMsg,NewBlockMsg的處理流程是藍色標記的區域,紅色區域是多帶帶的協程,是fetcher處理隊列中區塊的流程,如果從隊列中取出的區塊是當前鏈需要的,校驗后,調用blockchian.InsertChain()把區塊插入到區塊鏈,最后寫入數據庫,這是黃色部分。最后,綠色部分是NewBlockHashesMsg的處理流程,代碼流程上是比較復雜的,為了能通過圖描述整體流程,我把它簡化掉了。
仔細看看這幅圖,掌握整體的流程后,接下來看每個步驟的細節。
NewBlockMsg的處理本節介紹節點收到完整區塊的處理,流程如下:
首先進行RLP編解碼,然后標記發送消息的Peer已經知道這個區塊,這樣本節點最后廣播這個區塊的Hash時,不會再發送給該Peer。
將區塊存入到fetcher的隊列,調用fetcher.Enqueue。
更新Peer的Head位置,然后判斷本地鏈是否落后于Peer的鏈,如果是,則通過Peer更新本地鏈。
只看handle.Msg()的NewBlockMsg相關的部分。
case msg.Code == NewBlockMsg: // Retrieve and decode the propagated block // 收到新區塊,解碼,賦值接收數據 var request newBlockData if err := msg.Decode(&request); err != nil { return errResp(ErrDecode, "%v: %v", msg, err) } request.Block.ReceivedAt = msg.ReceivedAt request.Block.ReceivedFrom = p // Mark the peer as owning the block and schedule it for import // 標記peer知道這個區塊 p.MarkBlock(request.Block.Hash()) // 為啥要如隊列?已經得到完整的區塊了 // 答:存入fetcher的優先級隊列,fetcher會從隊列中選取當前高度需要的塊 pm.fetcher.Enqueue(p.id, request.Block) // Assuming the block is importable by the peer, but possibly not yet done so, // calculate the head hash and TD that the peer truly must have. // 截止到parent區塊的頭和難度 var ( trueHead = request.Block.ParentHash() trueTD = new(big.Int).Sub(request.TD, request.Block.Difficulty()) ) // Update the peers total difficulty if better than the previous // 如果收到的塊的難度大于peer之前的,以及自己本地的,就去和這個peer同步 // 問題:就只用了一下塊里的hash指,為啥不直接使用這個塊呢,如果這個塊不能用,干嘛不少發送些數據,減少網絡負載呢。 // 答案:實際上,這個塊加入到了優先級隊列中,當fetcher的loop檢查到當前下一個區塊的高度,正是隊列中有的,則不再向peer請求 // 該區塊,而是直接使用該區塊,檢查無誤后交給block chain執行insertChain if _, td := p.Head(); trueTD.Cmp(td) > 0 { p.SetHead(trueHead, trueTD) // Schedule a sync if above ours. Note, this will not fire a sync for a gap of // a singe block (as the true TD is below the propagated block), however this // scenario should easily be covered by the fetcher. currentBlock := pm.blockchain.CurrentBlock() if trueTD.Cmp(pm.blockchain.GetTd(currentBlock.Hash(), currentBlock.NumberU64())) > 0 { go pm.synchronise(p) } } //------------------------ 以上 handleMsg // Enqueue tries to fill gaps the the fetcher"s future import queue. // 發給inject通道,當前協程在handleMsg,通過通道發送給fetcher的協程處理 func (f *Fetcher) Enqueue(peer string, block *types.Block) error { op := &inject{ origin: peer, block: block, } select { case f.inject <- op: return nil case <-f.quit: return errTerminated } } //------------------------ 以下 fetcher.loop處理inject部分 case op := <-f.inject: // A direct block insertion was requested, try and fill any pending gaps // 區塊加入隊列,首先也填入未決的間距 propBroadcastInMeter.Mark(1) f.enqueue(op.origin, op.block) //------------------------ 如隊列函數 // enqueue schedules a new future import operation, if the block to be imported // has not yet been seen. // 把導入的新區塊放進來 func (f *Fetcher) enqueue(peer string, block *types.Block) { hash := block.Hash() // Ensure the peer isn"t DOSing us // 防止peer的DOS攻擊 count := f.queues[peer] + 1 if count > blockLimit { log.Debug("Discarded propagated block, exceeded allowance", "peer", peer, "number", block.Number(), "hash", hash, "limit", blockLimit) propBroadcastDOSMeter.Mark(1) f.forgetHash(hash) return } // Discard any past or too distant blocks // 高度檢查:未來太遠的塊丟棄 if dist := int64(block.NumberU64()) - int64(f.chainHeight()); dist < -maxUncleDist || dist > maxQueueDist { log.Debug("Discarded propagated block, too far away", "peer", peer, "number", block.Number(), "hash", hash, "distance", dist) propBroadcastDropMeter.Mark(1) f.forgetHash(hash) return } // Schedule the block for future importing // 塊先加入優先級隊列,加入鏈之前,還有很多要做 if _, ok := f.queued[hash]; !ok { op := &inject{ origin: peer, block: block, } f.queues[peer] = count f.queued[hash] = op f.queue.Push(op, -float32(block.NumberU64())) if f.queueChangeHook != nil { f.queueChangeHook(op.block.Hash(), true) } log.Debug("Queued propagated block", "peer", peer, "number", block.Number(), "hash", hash, "queued", f.queue.Size()) } }fetcher隊列處理
本節我們看看,區塊加入隊列后,fetcher如何處理區塊,為何不直接校驗區塊,插入到本地鏈?
由于以太坊又Uncle的機制,節點可能收到老一點的一些區塊。另外,節點可能由于網絡原因,落后了幾個區塊,所以可能收到“未來”的一些區塊,這些區塊都不能直接插入到本地鏈。
區塊入的隊列是一個優先級隊列,高度低的區塊會被優先取出來。fetcher.loop是多帶帶協程,不斷運轉,清理fecther中的事務和事件。首先會清理正在fetching的區塊,但已經超時。然后處理優先級隊列中的區塊,判斷高度是否是下一個區塊,如果是則調用f.insert()函數,校驗后調用BlockChain.InsertChain(),成功插入后,廣播新區塊的Hash。
// Loop is the main fetcher loop, checking and processing various notification // events. func (f *Fetcher) loop() { // Iterate the block fetching until a quit is requested fetchTimer := time.NewTimer(0) completeTimer := time.NewTimer(0) for { // Clean up any expired block fetches // 清理過期的區塊 for hash, announce := range f.fetching { if time.Since(announce.time) > fetchTimeout { f.forgetHash(hash) } } // Import any queued blocks that could potentially fit // 導入隊列中合適的塊 height := f.chainHeight() for !f.queue.Empty() { op := f.queue.PopItem().(*inject) hash := op.block.Hash() if f.queueChangeHook != nil { f.queueChangeHook(hash, false) } // If too high up the chain or phase, continue later // 塊不是鏈需要的下一個塊,再入優先級隊列,停止循環 number := op.block.NumberU64() if number > height+1 { f.queue.Push(op, -float32(number)) if f.queueChangeHook != nil { f.queueChangeHook(hash, true) } break } // Otherwise if fresh and still unknown, try and import // 高度正好是我們想要的,并且鏈上也沒有這個塊 if number+maxUncleDist < height || f.getBlock(hash) != nil { f.forgetBlock(hash) continue } // 那么,塊插入鏈 f.insert(op.origin, op.block) } //省略 } }
func (f *Fetcher) insert(peer string, block *types.Block) { hash := block.Hash() // Run the import on a new thread log.Debug("Importing propagated block", "peer", peer, "number", block.Number(), "hash", hash) go func() { defer func() { f.done <- hash }() // If the parent"s unknown, abort insertion parent := f.getBlock(block.ParentHash()) if parent == nil { log.Debug("Unknown parent of propagated block", "peer", peer, "number", block.Number(), "hash", hash, "parent", block.ParentHash()) return } // Quickly validate the header and propagate the block if it passes // 驗證區塊頭,成功后廣播區塊 switch err := f.verifyHeader(block.Header()); err { case nil: // All ok, quickly propagate to our peers propBroadcastOutTimer.UpdateSince(block.ReceivedAt) go f.broadcastBlock(block, true) case consensus.ErrFutureBlock: // Weird future block, don"t fail, but neither propagate default: // Something went very wrong, drop the peer log.Debug("Propagated block verification failed", "peer", peer, "number", block.Number(), "hash", hash, "err", err) f.dropPeer(peer) return } // Run the actual import and log any issues // 調用回調函數,實際是blockChain.insertChain if _, err := f.insertChain(types.Blocks{block}); err != nil { log.Debug("Propagated block import failed", "peer", peer, "number", block.Number(), "hash", hash, "err", err) return } // If import succeeded, broadcast the block propAnnounceOutTimer.UpdateSince(block.ReceivedAt) go f.broadcastBlock(block, false) // Invoke the testing hook if needed if f.importedHook != nil { f.importedHook(block) } }() }NewBlockHashesMsg的處理
本節介紹NewBlockHashesMsg的處理,其實,消息處理是簡單的,而復雜一點的是從Peer哪獲取完整的區塊,下節再看。
流程如下:
對消息進行RLP解碼,然后標記Peer已經知道此區塊。
尋找出本地區塊鏈不存在的區塊Hash值,把這些未知的Hash通知給fetcher。
fetcher.Notify記錄好通知信息,塞入notify通道,以便交給fetcher的協程。
fetcher.loop()會對notify中的消息進行處理,確認區塊并非DOS攻擊,然后檢查區塊的高度,判斷該區塊是否已經在fetching或者comleting(代表已經下載區塊頭,在下載body),如果都沒有,則加入到announced中,觸發0s定時器,進行處理。
關于announced下節再介紹。
// handleMsg()部分 case msg.Code == NewBlockHashesMsg: var announces newBlockHashesData if err := msg.Decode(&announces); err != nil { return errResp(ErrDecode, "%v: %v", msg, err) } // Mark the hashes as present at the remote node for _, block := range announces { p.MarkBlock(block.Hash) } // Schedule all the unknown hashes for retrieval // 把本地鏈沒有的塊hash找出來,交給fetcher去下載 unknown := make(newBlockHashesData, 0, len(announces)) for _, block := range announces { if !pm.blockchain.HasBlock(block.Hash, block.Number) { unknown = append(unknown, block) } } for _, block := range unknown { pm.fetcher.Notify(p.id, block.Hash, block.Number, time.Now(), p.RequestOneHeader, p.RequestBodies) }
// Notify announces the fetcher of the potential availability of a new block in // the network. // 通知fetcher(自己)有新塊產生,沒有塊實體,有hash、高度等信息 func (f *Fetcher) Notify(peer string, hash common.Hash, number uint64, time time.Time, headerFetcher headerRequesterFn, bodyFetcher bodyRequesterFn) error { block := &announce{ hash: hash, number: number, time: time, origin: peer, fetchHeader: headerFetcher, fetchBodies: bodyFetcher, } select { case f.notify <- block: return nil case <-f.quit: return errTerminated } }
// fetcher.loop()的notify通道消息處理 case notification := <-f.notify: // A block was announced, make sure the peer isn"t DOSing us propAnnounceInMeter.Mark(1) count := f.announces[notification.origin] + 1 if count > hashLimit { log.Debug("Peer exceeded outstanding announces", "peer", notification.origin, "limit", hashLimit) propAnnounceDOSMeter.Mark(1) break } // If we have a valid block number, check that it"s potentially useful // 高度檢查 if notification.number > 0 { if dist := int64(notification.number) - int64(f.chainHeight()); dist < -maxUncleDist || dist > maxQueueDist { log.Debug("Peer discarded announcement", "peer", notification.origin, "number", notification.number, "hash", notification.hash, "distance", dist) propAnnounceDropMeter.Mark(1) break } } // All is well, schedule the announce if block"s not yet downloading // 檢查是否已經在下載,已下載則忽略 if _, ok := f.fetching[notification.hash]; ok { break } if _, ok := f.completing[notification.hash]; ok { break } // 更新peer已經通知給我們的區塊數量 f.announces[notification.origin] = count // 把通知信息加入到announced,供調度 f.announced[notification.hash] = append(f.announced[notification.hash], notification) if f.announceChangeHook != nil && len(f.announced[notification.hash]) == 1 { f.announceChangeHook(notification.hash, true) } if len(f.announced) == 1 { // 有通知放入到announced,則重設0s定時器,loop的另外一個分支會處理這些通知 f.rescheduleFetch(fetchTimer) }fetcher獲取完整區塊
本節介紹fetcher獲取完整區塊的過程,這也是fetcher最重要的功能,會涉及到fetcher至少80%的代碼。多帶帶拉放一大節吧。
Fetcher的大頭Fetcher最主要的功能就是獲取完整的區塊,然后在合適的實際交給InsertChain去驗證和插入到本地區塊鏈。我們還是從宏觀入手,看Fetcher是如何工作的,一定要先掌握好宏觀,因為代碼層面上沒有這么清晰。
宏觀首先,看兩個節點是如何交互,獲取完整區塊,使用時序圖的方式看一下,見圖6,流程很清晰不再文字介紹。
再看下獲取區塊過程中,fetcher內部的狀態轉移,它使用狀態來記錄,要獲取的區塊在什么階段,見圖7。我稍微解釋一下:
收到NewBlockHashesMsg后,相關信息會記錄到announced,進入announced狀態,代表了本節點接收了消息。
announced由fetcher協程處理,經過校驗后,會向給他發送消息的Peer發送請求,請求該區塊的區塊頭,然后進入fetching狀態。
獲取區塊頭后,如果區塊頭表示沒有交易和uncle,則轉移到completing狀態,并且使用區塊頭合成完整的區塊,加入到queued優先級隊列。
獲取區塊頭后,如果區塊頭表示該區塊有交易和uncle,則轉移到fetched狀態,然后發送請求,請求交易和uncle,然后轉移到completing狀態。
收到交易和uncle后,使用頭、交易、uncle這3個信息,生成完整的區塊,加入到隊列queued。
微觀接下來就是從代碼角度看如何獲取完整區塊的流程了,有點多,看不懂的時候,再回顧下上面宏觀的介紹圖。
首先看Fetcher的定義,它存放了通信數據和狀態管理,撿加注釋的看,上文提到的狀態,里面都有。
// Fetcher is responsible for accumulating block announcements from various peers // and scheduling them for retrieval. // 積累塊通知,然后調度獲取這些塊 type Fetcher struct { // Various event channels // 收到區塊hash值的通道 notify chan *announce // 收到完整區塊的通道 inject chan *inject blockFilter chan chan []*types.Block // 過濾header的通道的通道 headerFilter chan chan *headerFilterTask // 過濾body的通道的通道 bodyFilter chan chan *bodyFilterTask done chan common.Hash quit chan struct{} // Announce states // Peer已經給了本節點多少區塊頭通知 announces map[string]int // Per peer announce counts to prevent memory exhaustion // 已經announced的區塊列表 announced map[common.Hash][]*announce // Announced blocks, scheduled for fetching // 正在fetching區塊頭的請求 fetching map[common.Hash]*announce // Announced blocks, currently fetching // 已經fetch到區塊頭,還差body的請求,用來獲取body fetched map[common.Hash][]*announce // Blocks with headers fetched, scheduled for body retrieval // 已經得到區塊頭的 completing map[common.Hash]*announce // Blocks with headers, currently body-completing // Block cache // queue,優先級隊列,高度做優先級 // queues,統計peer通告了多少塊 // queued,代表這個塊如隊列了, queue *prque.Prque // Queue containing the import operations (block number sorted) queues map[string]int // Per peer block counts to prevent memory exhaustion queued map[common.Hash]*inject // Set of already queued blocks (to dedupe imports) // Callbacks getBlock blockRetrievalFn // Retrieves a block from the local chain verifyHeader headerVerifierFn // Checks if a block"s headers have a valid proof of work,驗證區塊頭,包含了PoW驗證 broadcastBlock blockBroadcasterFn // Broadcasts a block to connected peers,廣播給peer chainHeight chainHeightFn // Retrieves the current chain"s height insertChain chainInsertFn // Injects a batch of blocks into the chain,插入區塊到鏈的函數 dropPeer peerDropFn // Drops a peer for misbehaving // Testing hooks announceChangeHook func(common.Hash, bool) // Method to call upon adding or deleting a hash from the announce list queueChangeHook func(common.Hash, bool) // Method to call upon adding or deleting a block from the import queue fetchingHook func([]common.Hash) // Method to call upon starting a block (eth/61) or header (eth/62) fetch completingHook func([]common.Hash) // Method to call upon starting a block body fetch (eth/62) importedHook func(*types.Block) // Method to call upon successful block import (both eth/61 and eth/62) }
NewBlockHashesMsg消息的處理前面的小節已經講過了,不記得可向前翻看。這里從announced的狀態處理說起。loop()中,fetchTimer超時后,代表了收到了消息通知,需要處理,會從announced中選擇出需要處理的通知,然后創建請求,請求區塊頭,由于可能有很多節點都通知了它某個區塊的Hash,所以隨機的從這些發送消息的Peer中選擇一個Peer,發送請求的時候,為每個Peer都創建了多帶帶的協程。
case <-fetchTimer.C: // At least one block"s timer ran out, check for needing retrieval // 有區塊通知,去處理 request := make(map[string][]common.Hash) for hash, announces := range f.announced { if time.Since(announces[0].time) > arriveTimeout-gatherSlack { // Pick a random peer to retrieve from, reset all others // 可能有很多peer都發送了這個區塊的hash值,隨機選擇一個peer announce := announces[rand.Intn(len(announces))] f.forgetHash(hash) // If the block still didn"t arrive, queue for fetching // 本地還沒有這個區塊,創建獲取區塊的請求 if f.getBlock(hash) == nil { request[announce.origin] = append(request[announce.origin], hash) f.fetching[hash] = announce } } } // Send out all block header requests // 把所有的request發送出去 // 為每一個peer都創建一個協程,然后請求所有需要從該peer獲取的請求 for peer, hashes := range request { log.Trace("Fetching scheduled headers", "peer", peer, "list", hashes) // Create a closure of the fetch and schedule in on a new thread fetchHeader, hashes := f.fetching[hashes[0]].fetchHeader, hashes go func() { if f.fetchingHook != nil { f.fetchingHook(hashes) } for _, hash := range hashes { headerFetchMeter.Mark(1) fetchHeader(hash) // Suboptimal, but protocol doesn"t allow batch header retrievals } }() } // Schedule the next fetch if blocks are still pending f.rescheduleFetch(fetchTimer)
從Notify的調用中,可以看出,fetcherHeader()的實際函數是RequestOneHeader(),該函數使用的消息是GetBlockHeadersMsg,可以用來請求多個區塊頭,不過fetcher只請求一個。
pm.fetcher.Notify(p.id, block.Hash, block.Number, time.Now(), p.RequestOneHeader, p.RequestBodies) // RequestOneHeader is a wrapper around the header query functions to fetch a // single header. It is used solely by the fetcher. func (p *peer) RequestOneHeader(hash common.Hash) error { p.Log().Debug("Fetching single header", "hash", hash) return p2p.Send(p.rw, GetBlockHeadersMsg, &getBlockHeadersData{Origin: hashOrNumber{Hash: hash}, Amount: uint64(1), Skip: uint64(0), Reverse: false}) }
GetBlockHeadersMsg的處理如下:因為它是獲取多個區塊頭的,所以處理起來比較“麻煩”,還好,fetcher只獲取一個區塊頭,其處理在20行~33行,獲取下一個區塊頭的處理邏輯,這里就不看了,最后調用SendBlockHeaders()將區塊頭發送給請求的節點,消息是BlockHeadersMsg。
// handleMsg() // Block header query, collect the requested headers and reply case msg.Code == GetBlockHeadersMsg: // Decode the complex header query var query getBlockHeadersData if err := msg.Decode(&query); err != nil { return errResp(ErrDecode, "%v: %v", msg, err) } hashMode := query.Origin.Hash != (common.Hash{}) // Gather headers until the fetch or network limits is reached // 收集區塊頭,直到達到限制 var ( bytes common.StorageSize headers []*types.Header unknown bool ) // 自己已知區塊 && 少于查詢的數量 && 大小小于2MB && 小于能下載的最大數量 for !unknown && len(headers) < int(query.Amount) && bytes < softResponseLimit && len(headers) < downloader.MaxHeaderFetch { // Retrieve the next header satisfying the query // 獲取區塊頭 var origin *types.Header if hashMode { // fetcher 使用的模式 origin = pm.blockchain.GetHeaderByHash(query.Origin.Hash) } else { origin = pm.blockchain.GetHeaderByNumber(query.Origin.Number) } if origin == nil { break } number := origin.Number.Uint64() headers = append(headers, origin) bytes += estHeaderRlpSize // Advance to the next header of the query // 下一個區塊頭的獲取,不同策略,方式不同 switch { case query.Origin.Hash != (common.Hash{}) && query.Reverse: // ... } } return p.SendBlockHeaders(headers)
BlockHeadersMsg的處理很有意思,因為GetBlockHeadersMsg并不是fetcher獨占的消息,downloader也可以調用,所以,響應消息的處理需要分辨出是fetcher請求的,還是downloader請求的。它的處理邏輯是:fetcher先過濾收到的區塊頭,如果fetcher不要的,那就是downloader的,在調用fetcher.FilterHeaders的時候,fetcher就將自己要的區塊頭拿走了。
// handleMsg() case msg.Code == BlockHeadersMsg: // A batch of headers arrived to one of our previous requests var headers []*types.Header if err := msg.Decode(&headers); err != nil { return errResp(ErrDecode, "msg %v: %v", msg, err) } // If no headers were received, but we"re expending a DAO fork check, maybe it"s that // 檢查是不是當前DAO的硬分叉 if len(headers) == 0 && p.forkDrop != nil { // Possibly an empty reply to the fork header checks, sanity check TDs verifyDAO := true // If we already have a DAO header, we can check the peer"s TD against it. If // the peer"s ahead of this, it too must have a reply to the DAO check if daoHeader := pm.blockchain.GetHeaderByNumber(pm.chainconfig.DAOForkBlock.Uint64()); daoHeader != nil { if _, td := p.Head(); td.Cmp(pm.blockchain.GetTd(daoHeader.Hash(), daoHeader.Number.Uint64())) >= 0 { verifyDAO = false } } // If we"re seemingly on the same chain, disable the drop timer if verifyDAO { p.Log().Debug("Seems to be on the same side of the DAO fork") p.forkDrop.Stop() p.forkDrop = nil return nil } } // Filter out any explicitly requested headers, deliver the rest to the downloader // 過濾是不是fetcher請求的區塊頭,去掉fetcher請求的區塊頭再交給downloader filter := len(headers) == 1 if filter { // If it"s a potential DAO fork check, validate against the rules // 檢查是否硬分叉 if p.forkDrop != nil && pm.chainconfig.DAOForkBlock.Cmp(headers[0].Number) == 0 { // Disable the fork drop timer p.forkDrop.Stop() p.forkDrop = nil // Validate the header and either drop the peer or continue if err := misc.VerifyDAOHeaderExtraData(pm.chainconfig, headers[0]); err != nil { p.Log().Debug("Verified to be on the other side of the DAO fork, dropping") return err } p.Log().Debug("Verified to be on the same side of the DAO fork") return nil } // Irrelevant of the fork checks, send the header to the fetcher just in case // 使用fetcher過濾區塊頭 headers = pm.fetcher.FilterHeaders(p.id, headers, time.Now()) } // 剩下的區塊頭交給downloader if len(headers) > 0 || !filter { err := pm.downloader.DeliverHeaders(p.id, headers) if err != nil { log.Debug("Failed to deliver headers", "err", err) } }
FilterHeaders()是一個很有大智慧的函數,看起來耐人尋味,但實在妙。它要把所有的區塊頭,都傳遞給fetcher協程,還要獲取fetcher協程處理后的結果。fetcher.headerFilter是存放通道的通道,而filter是存放包含區塊頭過濾任務的通道。它先把filter傳遞給了headerFilter,這樣fetcher協程就在另外一段等待了,而后將headerFilterTask傳入filter,fetcher就能讀到數據了,處理后,再將數據寫回filter而剛好被FilterHeaders函數處理了,該函數實際運行在handleMsg()的協程中。
每個Peer都會分配一個ProtocolManager然后處理該Peer的消息,但fetcher只有一個事件處理協程,如果不創建一個filter,fetcher哪知道是誰發給它的區塊頭呢?過濾之后,該如何發回去呢?
// FilterHeaders extracts all the headers that were explicitly requested by the fetcher, // returning those that should be handled differently. // 尋找出fetcher請求的區塊頭 func (f *Fetcher) FilterHeaders(peer string, headers []*types.Header, time time.Time) []*types.Header { log.Trace("Filtering headers", "peer", peer, "headers", len(headers)) // Send the filter channel to the fetcher // 任務通道 filter := make(chan *headerFilterTask) select { // 任務通道發送到這個通道 case f.headerFilter <- filter: case <-f.quit: return nil } // Request the filtering of the header list // 創建過濾任務,發送到任務通道 select { case filter <- &headerFilterTask{peer: peer, headers: headers, time: time}: case <-f.quit: return nil } // Retrieve the headers remaining after filtering // 從任務通道,獲取過濾的結果并返回 select { case task := <-filter: return task.headers case <-f.quit: return nil } }
接下來要看f.headerFilter的處理,這段代碼有90行,它做了一下幾件事:
從f.headerFilter取出filter,然后取出過濾任務task。
它把區塊頭分成3類:unknown這不是分是要返回給調用者的,即handleMsg(), incomplete存放還需要獲取body的區塊頭,complete存放只包含區塊頭的區塊。遍歷所有的區塊頭,填到到對應的分類中,具體的判斷可看18行的注釋,記住宏觀中將的狀態轉移圖。
把unknonw中的區塊返回給handleMsg()。
把 incomplete的區塊頭獲取狀態移動到fetched狀態,然后觸發定時器,以便去處理complete的區塊。
把compelete的區塊加入到queued。
// fetcher.loop() case filter := <-f.headerFilter: // Headers arrived from a remote peer. Extract those that were explicitly // requested by the fetcher, and return everything else so it"s delivered // to other parts of the system. // 收到從遠端節點發送的區塊頭,過濾出fetcher請求的 // 從任務通道獲取過濾任務 var task *headerFilterTask select { case task = <-filter: case <-f.quit: return } headerFilterInMeter.Mark(int64(len(task.headers))) // Split the batch of headers into unknown ones (to return to the caller), // known incomplete ones (requiring body retrievals) and completed blocks. // unknown的不是fetcher請求的,complete放沒有交易和uncle的區塊,有頭就夠了,incomplete放 // 還需要獲取uncle和交易的區塊 unknown, incomplete, complete := []*types.Header{}, []*announce{}, []*types.Block{} // 遍歷所有收到的header for _, header := range task.headers { hash := header.Hash() // Filter fetcher-requested headers from other synchronisation algorithms // 是正在獲取的hash,并且對應請求的peer,并且未fetched,未completing,未queued if announce := f.fetching[hash]; announce != nil && announce.origin == task.peer && f.fetched[hash] == nil && f.completing[hash] == nil && f.queued[hash] == nil { // If the delivered header does not match the promised number, drop the announcer // 高度校驗,竟然不匹配,擾亂秩序,peer肯定是壞蛋。 if header.Number.Uint64() != announce.number { log.Trace("Invalid block number fetched", "peer", announce.origin, "hash", header.Hash(), "announced", announce.number, "provided", header.Number) f.dropPeer(announce.origin) f.forgetHash(hash) continue } // Only keep if not imported by other means // 本地鏈沒有當前區塊 if f.getBlock(hash) == nil { announce.header = header announce.time = task.time // If the block is empty (header only), short circuit into the final import queue // 如果區塊沒有交易和uncle,加入到complete if header.TxHash == types.DeriveSha(types.Transactions{}) && header.UncleHash == types.CalcUncleHash([]*types.Header{}) { log.Trace("Block empty, skipping body retrieval", "peer", announce.origin, "number", header.Number, "hash", header.Hash()) block := types.NewBlockWithHeader(header) block.ReceivedAt = task.time complete = append(complete, block) f.completing[hash] = announce continue } // Otherwise add to the list of blocks needing completion // 否則就是不完整的區塊 incomplete = append(incomplete, announce) } else { log.Trace("Block already imported, discarding header", "peer", announce.origin, "number", header.Number, "hash", header.Hash()) f.forgetHash(hash) } } else { // Fetcher doesn"t know about it, add to the return list // 沒請求過的header unknown = append(unknown, header) } } // 把未知的區塊頭,再傳遞會filter headerFilterOutMeter.Mark(int64(len(unknown))) select { case filter <- &headerFilterTask{headers: unknown, time: task.time}: case <-f.quit: return } // Schedule the retrieved headers for body completion // 把未完整的區塊加入到fetched,跳過已經在completeing中的,然后觸發completeTimer定時器 for _, announce := range incomplete { hash := announce.header.Hash() if _, ok := f.completing[hash]; ok { continue } f.fetched[hash] = append(f.fetched[hash], announce) if len(f.fetched) == 1 { f.rescheduleComplete(completeTimer) } } // Schedule the header-only blocks for import // 把只有頭的區塊入隊列 for _, block := range complete { if announce := f.completing[block.Hash()]; announce != nil { f.enqueue(announce.origin, block) } }
跟隨狀態圖的轉義,剩下的工作是fetched轉移到completing,上面的流程已經觸發了completeTimer定時器,超時后就會處理,流程與請求Header類似,不再贅述,此時發送的請求消息是GetBlockBodiesMsg,實際調的函數是RequestBodies。
// fetcher.loop() case <-completeTimer.C: // At least one header"s timer ran out, retrieve everything // 至少有1個header已經獲取完了 request := make(map[string][]common.Hash) // 遍歷所有待獲取body的announce for hash, announces := range f.fetched { // Pick a random peer to retrieve from, reset all others // 隨機選一個Peer發送請求,因為可能已經有很多Peer通知它這個區塊了 announce := announces[rand.Intn(len(announces))] f.forgetHash(hash) // If the block still didn"t arrive, queue for completion // 如果本地沒有這個區塊,則放入到completing,創建請求 if f.getBlock(hash) == nil { request[announce.origin] = append(request[announce.origin], hash) f.completing[hash] = announce } } // Send out all block body requests // 發送所有的請求,獲取body,依然是每個peer一個多帶帶協程 for peer, hashes := range request { log.Trace("Fetching scheduled bodies", "peer", peer, "list", hashes) // Create a closure of the fetch and schedule in on a new thread if f.completingHook != nil { f.completingHook(hashes) } bodyFetchMeter.Mark(int64(len(hashes))) go f.completing[hashes[0]].fetchBodies(hashes) } // Schedule the next fetch if blocks are still pending f.rescheduleComplete(completeTimer)
handleMsg()處理該消息也是干凈利落,直接獲取RLP格式的body,然后發送響應消息。
// handleMsg() case msg.Code == GetBlockBodiesMsg: // Decode the retrieval message msgStream := rlp.NewStream(msg.Payload, uint64(msg.Size)) if _, err := msgStream.List(); err != nil { return err } // Gather blocks until the fetch or network limits is reached var ( hash common.Hash bytes int bodies []rlp.RawValue ) // 遍歷所有請求 for bytes < softResponseLimit && len(bodies) < downloader.MaxBlockFetch { // Retrieve the hash of the next block if err := msgStream.Decode(&hash); err == rlp.EOL { break } else if err != nil { return errResp(ErrDecode, "msg %v: %v", msg, err) } // Retrieve the requested block body, stopping if enough was found // 獲取body,RLP格式 if data := pm.blockchain.GetBodyRLP(hash); len(data) != 0 { bodies = append(bodies, data) bytes += len(data) } } return p.SendBlockBodiesRLP(bodies)
響應消息BlockBodiesMsg的處理與處理獲取header的處理原理相同,先交給fetcher過濾,然后剩下的才是downloader的。需要注意一點,響應消息里只包含交易列表和叔塊列表。
// handleMsg() case msg.Code == BlockBodiesMsg: // A batch of block bodies arrived to one of our previous requests var request blockBodiesData if err := msg.Decode(&request); err != nil { return errResp(ErrDecode, "msg %v: %v", msg, err) } // Deliver them all to the downloader for queuing // 傳遞給downloader去處理 transactions := make([][]*types.Transaction, len(request)) uncles := make([][]*types.Header, len(request)) for i, body := range request { transactions[i] = body.Transactions uncles[i] = body.Uncles } // Filter out any explicitly requested bodies, deliver the rest to the downloader // 先讓fetcher過濾去fetcher請求的body,剩下的給downloader filter := len(transactions) > 0 || len(uncles) > 0 if filter { transactions, uncles = pm.fetcher.FilterBodies(p.id, transactions, uncles, time.Now()) } // 剩下的body交給downloader if len(transactions) > 0 || len(uncles) > 0 || !filter { err := pm.downloader.DeliverBodies(p.id, transactions, uncles) if err != nil { log.Debug("Failed to deliver bodies", "err", err) } }
過濾函數的原理也與Header相同。
// FilterBodies extracts all the block bodies that were explicitly requested by // the fetcher, returning those that should be handled differently. // 過去出fetcher請求的body,返回它沒有處理的,過程類型header的處理 func (f *Fetcher) FilterBodies(peer string, transactions [][]*types.Transaction, uncles [][]*types.Header, time time.Time) ([][]*types.Transaction, [][]*types.Header) { log.Trace("Filtering bodies", "peer", peer, "txs", len(transactions), "uncles", len(uncles)) // Send the filter channel to the fetcher filter := make(chan *bodyFilterTask) select { case f.bodyFilter <- filter: case <-f.quit: return nil, nil } // Request the filtering of the body list select { case filter <- &bodyFilterTask{peer: peer, transactions: transactions, uncles: uncles, time: time}: case <-f.quit: return nil, nil } // Retrieve the bodies remaining after filtering select { case task := <-filter: return task.transactions, task.uncles case <-f.quit: return nil, nil } }
實際過濾body的處理瞧一下,這和Header的處理是不同的。直接看不點:
它要的區塊,多帶帶取出來存到blocks中,它不要的繼續留在task中。
判斷是不是fetcher請求的方法:如果交易列表和叔塊列表計算出的hash值與區塊頭中的一樣,并且消息來自請求的Peer,則就是fetcher請求的。
將blocks中的區塊加入到queued,終結。
case filter := <-f.bodyFilter: // Block bodies arrived, extract any explicitly requested blocks, return the rest var task *bodyFilterTask select { case task = <-filter: case <-f.quit: return } bodyFilterInMeter.Mark(int64(len(task.transactions))) blocks := []*types.Block{} // 獲取的每個body的txs列表和uncle列表 // 遍歷每個區塊的txs列表和uncle列表,計算hash后判斷是否是當前fetcher請求的body for i := 0; i < len(task.transactions) && i < len(task.uncles); i++ { // Match up a body to any possible completion request matched := false // 遍歷所有保存的請求,因為tx和uncle,不知道它是屬于哪個區塊的,只能去遍歷所有的請求,通常量不大,所以遍歷沒有性能影響 for hash, announce := range f.completing { if f.queued[hash] == nil { // 把傳入的每個塊的hash和unclehash和它請求出去的記錄進行對比,匹配則說明是fetcher請求的區塊body txnHash := types.DeriveSha(types.Transactions(task.transactions[i])) uncleHash := types.CalcUncleHash(task.uncles[i]) if txnHash == announce.header.TxHash && uncleHash == announce.header.UncleHash && announce.origin == task.peer { // Mark the body matched, reassemble if still unknown matched = true // 如果當前鏈還沒有這個區塊,則收集這個區塊,合并成新區塊 if f.getBlock(hash) == nil { block := types.NewBlockWithHeader(announce.header).WithBody(task.transactions[i], task.uncles[i]) block.ReceivedAt = task.time blocks = append(blocks, block) } else { f.forgetHash(hash) } } } } // 從task中移除fetcher請求的數據 if matched { task.transactions = append(task.transactions[:i], task.transactions[i+1:]...) task.uncles = append(task.uncles[:i], task.uncles[i+1:]...) i-- continue } } // 將剩余的數據返回 bodyFilterOutMeter.Mark(int64(len(task.transactions))) select { case filter <- task: case <-f.quit: return } // Schedule the retrieved blocks for ordered import // 把收集的區塊加入到隊列 for _, block := range blocks { if announce := f.completing[block.Hash()]; announce != nil { f.enqueue(announce.origin, block) } } }
至此,fetcher獲取完整區塊的流程講完了,fetcher模塊中80%的代碼也都貼出來了,還有2個值得看看的函數:
forgetHash(hash common.Hash) :用于清空指定hash指的記/狀態錄信息。
forgetBlock(hash common.Hash):用于從隊列中移除一個區塊。
最后了,再回到開始看看fetcher模塊和新區塊的傳播流程,有沒有豁然開朗。
文章版權歸作者所有,未經允許請勿轉載,若此文章存在違規行為,您可以聯系管理員刪除。
轉載請注明本文地址:http://specialneedsforspecialkids.com/yun/24395.html
摘要:前言是以太坊封定義的一個接口,它的功能可以分為類驗證區塊類,主要用在將區塊加入到區塊鏈前,對區塊進行共識驗證。輔助類生成以太坊共識相關的。被使用,是以太坊狀態管理服務,當報告數據的時候,需要獲取區塊的信息。 前言 engine是以太坊封定義的一個接口,它的功能可以分為3類: 驗證區塊類,主要用在將區塊加入到區塊鏈前,對區塊進行共識驗證。 產生區塊類,主要用在挖礦時。 輔助類。 接下...
摘要:接下來我們將從以下角度介紹礦工角色。我們分別使用礦長副礦長礦工進行類比。副礦長,負責具體挖礦工作的安排,把挖礦任務安排給。礦工的主要函數介紹和的主要函數,他們是礦工的具體運作機制。負責處理外部事件。 前言 礦工在PoW中負責了產生區塊的工作,把一大堆交易交給它,它生成一個證明自己做了很多工作的區塊,然后將這個區塊加入到本地區塊鏈并且廣播給其他節點。 接下來我們將從以下角度介紹礦工: ...
摘要:下面來看看具體是怎么實現接口的可以看到,啟動了多個線程調用函數,當有線程挖到時,會通過傳入的通道傳出結果??梢钥吹皆谥饕h中,不斷遞增的值,調用函數計算上面公式中的左邊,而則是公式的右邊。 前言 挖礦(mine)是指礦工節點互相競爭生成新區塊以寫入整個區塊鏈獲得獎勵的過程.共識(consensus)是指區塊鏈各個節點對下一個區塊的內容形成一致的過程在以太坊中, miner包向外提供挖...
摘要:是以太坊中存儲區塊數據的核心數據結構,它和融合一個樹形結構,理解結構對之后學習以太坊區塊以及智能合約狀態存儲結構的模塊源碼很有幫助。 MPT(Merkle Patricia Tries)是以太坊中存儲區塊數據的核心數據結構,它Merkle Tree和Patricia Tree融合一個樹形結構,理解MPT結構對之后學習以太坊區塊header以及智能合約狀態存儲結構的模塊源碼很有幫助。 首...
閱讀 3483·2021-11-18 10:02
閱讀 1612·2021-10-12 10:12
閱讀 2990·2021-10-09 09:53
閱讀 4858·2021-09-09 09:34
閱讀 847·2021-09-06 15:02
閱讀 2777·2021-08-05 10:02
閱讀 3134·2019-08-30 15:44
閱讀 3121·2019-08-28 18:04