Skip to content

nem0z/bitcoin-crawler

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

79 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Component

  1. Crawler This is the core component of the application, basically an engine which start new connections and handle their result. There is only 1 per instance

  2. Node A Node describe a node in the Bitcoin network and keep track of some information about it:

type Node struct {
	Timestamp time.Time `json:"timestamp"`
	Info      *Info     `json:"info"`
	Addr      *Addr     `json:"addr"`
	Ping      bool      `json:"ping"`
}
  1. Peer A peer is bascially a connection to a Bitcoin node, it's used to communicate with this node
type Peer struct {
	ip        string
	port      int
	conn      net.Conn
	handlers  Handlers
	Info      *Info
	Addrs     []*Addr
	PingNonce []byte
	PingAt    time.Time
	PongAt    time.Time
	queue     chan *message.Message
	onClose   chan *Node
}

Workflow

  1. Main
  • Init database
  • Load known peers
  • Start crawlers with known peers
  • Load OK peers from database
  • Wait for interupt (SIGINT or SIGTERM)
  1. On Interupt
  • Export nodes to /expot/nodes.json
  • Save / Update nodes in DB
  1. Create a crawler
  • Create a struct crawler
  • Register all the provided addr to the crawler
  • Start the ResultHandler -> Basically update the status of a Node in map nodes
  • Start N worker to handle new nodes
  • Start moniroting -> Basically a go routine which log the state of the map of nodes every 10s
  1. Worker
  • Verify if the Node is already known (if yes stop here)
  • Create + setup the new Peer
  • Start the Peer
  • Send 10 GetAddr message to the Peer
  • After 60s close the connection (should be reworked to use context)
  • Add the peer (node) to the map of known nodes
  1. Create a new Peer
  • Create a tcp connection to the IPv4 or IPv6 address
  • Init the struct
  1. Start a Peer
  • Send Version message
  • Send Verack message
  • Send Ping message
  • Send GetAddr message
  • Start handler for differents messages

About

Go implementation of a bitcoin crawler

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages