-
Notifications
You must be signed in to change notification settings - Fork 7
Fullnode
By Lululute
- Introduction
- Disclaimer
- Picking Parts
- Assembly
- Configuring Ubuntu
- Syncing Geth
- Docker and Truebit OS
When I tried verifying, I noticed I was getting a lot of timeout errors that prevented me from completing verification. To encourage timely execution of tasks, Truebit OS imposes a time limit on each step of a task. If one of the task participants fails to report, they will “time out” and be removed from the task.
There could be all sorts of reasons for why this happens. One possibility is the latency of the light client. By default, the Truebit container connects to mainnet with a light client of Geth. Light nodes receive and send transactions by way of a full node intermediary. Given the traffic on mainnet, it’s possible that verifiers’ challenges occasionally just aren’t being transferred in time.
I don’t know yet if this is the reason why verifiers sometimes get dropped. On this page I’ll be tracking my own performance as a full node verifier and the number of dropped tasks I get, which will hopefully give a better sense of whether a full node is necessary for solvers/verifiers. I encourage you to share your experiences with light and full nodes running Truebit, so we can all help create better node infrastructure.
UPDATE: While I've encountered other errors during operation (which I will try to document in a separate post), I can report that after setting up the full node, as long as gas is appropriately high, I have yet to "time out" of a task due to lagging transactions.
If you’d like to know how I set up my own full node, read on:
This document is primarily about building a new computer and running a full node with Ubuntu 20.04.2 and Geth. You do not need to build a computer to run a node; some prebuilt PCs have the right specs, and there are commercially available devices like Avado, which are mini-computers loaded with an OS purpose-built for running Ethereum. You could also build smaller, cheap devices (such as an SSD plugged into a Raspberry Pi) that are capable of running a node, but I can’t attest to their performance.
I am not an expert in any of these subjects. All of the information here has been pieced together from search engines and the accumulated knowledge of the Ethereum, Truebit, and Linux communities, so I apologize in advance if there are redundancies, errors, or other head-scratchers in the code snippets and procedures I provide. If there are any errors you’d like to fix, or if you have improvements to suggest, please do so.
If you’re interested in solving/verifying but totally new to Truebit, I recommend following the quickstart guide first and running a node on Goerli so you know what everything looks like.
https://hub.docker.com/r/truebitprotocol/truebit-eth
If you’ve never built a computer before, it helps to know what you need. A basic PC requires a Motherboard, CPU, CPU Cooler, RAM, SSD, PSU, Case, and Peripherals. Generally speaking, while you don’t want to overpay, it’s a good idea to save yourself a headache and buy quality components. You’ll be building something you intend to run 24/7.
For our purposes, the most important component is the SSD, and the correct kind of SSD. Solid state drives are required for Ethereum syncing because of the tremendous amount of data that needs to be quickly written to the drive, and the constant writing that will go on once you’ve synced your node. You’ll need plenty of storage space, as the chain is large and will continue to grow.
SSDs store data using specialized transistors. Much SSD development has focused on the number of memory cells per transistor; increasing the number of cells gives greater storage in the same amount of space. This reduces the number of transistors needed in a given device, thereby reducing cost.
The tradeoff is reduced performance and longevity, as the errors introduced by multiple cells need to be corrected, which decreases write speeds. For many purposes this is fine, but because you want sustained writes and a long life, you’ll want to get an MLC (multi-level cell, e.g. double-level) or TLC (triple-level-cell) drive. SLCs also exist, but I can’t comment on how they compare to the others; I opted for an MLC. Avoid QLC drives.
Some drives have an “SLC cache” used for writing, but this won’t work for sustained writing, because once the cache is full, it will go back to writing on the much slower main drive.
You’ll also want sufficient RAM and a solid multicore CPU. The Ethereum chain is extremely large and dense, and more RAM and cores will improve syncing speed. They should also reduce latency once you’ve synced. RAM comes in varying speeds, and you’ll probably want yours to be relatively fast.
All of your parts need to be mutually compatible. The CPU socket of your Motherboard, for example, will only support certain types of CPUs, and both your CPU and Motherboard need to be compatible with your RAM. Motherboards come in different sizes and your case needs to be large enough to hold it. Your Motherboard also needs a slot for the SSD. Always read the specs for components and make sure that they all fit together. It doesn’t hurt to quadruple-check.
You can use a site like pcpartpicker to help you ascertain compatibilities, and also estimate how much power your PSU needs to supply. The amount of power needed to run your machine depends on its components; without a graphics card the power draw won’t be terribly high, but if you intend to add a GPU later, you can always get a bit more wattage than you need. Get a PSU that is at least silver or gold medal rated.
You also might consider an uninterruptible power supply to handle both your computer and router, which will give you time to shut down your node in the event of power failure. Sudden termination of a Geth session can corrupt your chaindata, requiring you to resync the entire chain.
For my device I opted for:
- 32 GB RAM DDR4 3200 MHz
- 2.7 gHz 8-Core Processor
- 2 TB NVMe MLC SSD
This is probably overkill for just running Truebit OS. I’m also not knowledgeable enough to know where I’m getting bottlenecks and which components are limiting which. With these components I was able to fast-sync my Geth node in about 44 hours.
Building the computer is somewhat of a surgical procedure. It can help greatly to have a second person. It is mandatory to watch some videos and read the instructions that come with your components. Clean your work area (and yourself) and remember that you are handling sensitive electronics. Before and during work, periodically discharge yourself of static electricity by touching a grounded metal object. You can also wear a grounding armband. Try to touch the components as little as possible.
My general order of operations is to unpack all components, set the Motherboard on its box, install the CPU and cooler, then install the SSD and RAM.
Once I’ve assembled a computer, I check that all the components work by connecting the PSU and a monitor and turning on the Motherboard without electrocuting myself. If there are problems, I’ll need to diagnose which part has failed, send it back, and get a new one.
If everything works, I turn off and unplug the PSU, snap the I/O shield into the case, and lower the assembled computer onto the standups inside the case. Then it’s time to screw the Motherboard into place, not too tightly. Connect the case fans, case cables, and PSU lines to the Motherboard. Use good cable management, keeping in mind that the case needs to be well-ventilated. Turn on the computer again to make sure that the fans are running and that the power and reset buttons work properly. Then screw in the PSU and close the case.
For this build I chose Ubuntu 20.04.2, the latest stable release. I installed it using a Windows application called Rufus, which takes an ISO image and mounts it on a USB drive. Follow this guide:
https://ubuntu.com/tutorials/create-a-usb-stick-on-windows#2-requirements
Your new computer should automatically boot from the USB, and you’ll be prompted to install Ubuntu. Pick a cool username and a good password, and write it down somewhere. You’ll need to enter it frequently. After installation you’ll be prompted to enable Livepatch, which is a service that pushes critical security updates without requiring a system restart. Since the node will be running continuously, it seemed like something I’d want, so I went ahead and turned it on.
From here on you’ll be diving heavily into the Terminal, which is where your node will eventually live. You can find the Terminal by clicking the grid icon in the lower left corner.
The first step will be enabling UFW, or Uncomplicated Firewall. We’ll start right away with the sudo command, or “super-do”, behind which is locked your system’s “root” (administrator) privileges. You will be required to enter your password when you use it.
sudo apt-get install ufw
then enter:
sudo ufw enable
then enter:
sudo ufw default deny incoming
sudo ufw default allow outgoing
This blocks all incoming connections. This won’t make you immune to hacking, but it does make your computer much more secure. You won’t, however, be able to connect to Geth.
sudo ufw allow 30303
This opens port 30303, which is the port used by the Ethereum network and as of now should be the only port you need to run Geth and use Truebit OS. I know that IPFS plays some role in the Truebit filesystem, but I don’t know whether it requires a certain port. This command should open both the TCP and UDP ports, but if for some reason it doesn’t quite work, you can also do:
sudo ufw allow 30303/tcp
sudo ufw allow 30303/udp
You can check the state of your UFW settings with the command:
sudo ufw status verbose
Another, more minor change: click the upper right corner of your screen to open a dropdown menu. Click Settings. Click Privacy. Disable Connectivity Checking.
The next step is installing Geth. I followed the instructions on the Ethereum Foundation’s website, which describes how to install using the Terminal:
https://geth.ethereum.org/docs/install-and-build/installing-geth
A note here on RPCs, or remote procedure calls. When I was reading about node security I came across old stories of nodes getting hacked via RPC. By default, the Geth RPC will only accept calls from the machine running Geth, which means by default only your local computer will be able to make RPC calls to your node. You won’t need to fool with your RPC settings to connect to Geth or Truebit OS, so just leave them alone.
Let me start off by mentioning a point of fact from the Truebit quickstart guide: if you are running Truebit OS in a docker container, it will only work if Geth is running in the same container. You could conceivably sync your node inside the Truebit container itself. I genuinely don’t know if this is a good idea, so I didn’t try it.
A note now on sync modes, specifically “fast” versus “full”. While both sync types produce what are commonly referred to as “full nodes”, there is an important distinction between fast sync and full sync. A full (archive) sync records all transaction data for every transaction that has ever occurred on the blockchain, whereas a fast sync simply checks whether all the recorded data is valid. A fast synced node cryptographically comports with the consensus version of Ethereum, but does not have the detailed transaction data of a full archive node.
A fast node can still act as a validator and, since it has a cryptographically correct copy of the blockchain, it can identify when a full archive node tries to cheat. Note that once a fast-synced node completes synchronization, thereafter it will act as a full archive node, recording all transaction data as it happens.
I decided to opt for fast mode. If you want to run a full archive node, you’ll likely want more storage space, and you should be prepared for a very long sync time.
When initializing the sync, I entered the following command:
geth --syncmode fast --cache 10240
Fast is the default syncmode, but I felt like entering it anyway. The important command here is --cache, as the default geth cache is 1024 (1 GB of RAM). Naturally, bumping it up to 10240 gives us a cache of 10 GB. A bigger cache will improve syncing speed. With 32 GB of RAM you could probably set an even higher cache value, but keep in mind that you need an amount of RAM equal to the cache size to serve as a “garbage collector”, and you need some RAM to run the operating system. 10 GB seemed like a comfortable amount.
Syncing will now begin. I started mine at around 2:40 PM, and after a few minutes of what appeared to be a stall, with warnings and dropped peers and failed synchronizations, there then came a flood of imported block headers, state entries, and block receipts. By 6:40 AM the next morning, importation had ended and state snapshots had begun. At 10:05 AM the following day, the node entered the download queue, and at 10:31 AM it wrote a clean trie cache to disk, regenerated a local transaction journal, and persisted the clean trie cache.
You can check the state of your sync by opening a second Terminal window, and typing:
geth attach
to open the Geth console. You can then enter:
eth.syncing
If it returns “false”, then you are synced.
At this point, in the Geth window you will see a continual stream of ‘Imported new chain segment’, ‘Unindexed transactions’, and ‘Deep froze chain segment’, with occasional errors and dropped peers. Note that if you ever need to close your node, hit CTRL+C and wait until the node has wound down and fully exited the blockchain. As mentioned above, if you suddenly terminate your Geth session, you can corrupt your chaindata.
Since you synced your node outside of the container, you will now need to shut it down.
Next, in a new Terminal window, install Docker by following the instructions on their site:
https://docs.docker.com/engine/install/ubuntu/
Then go to the Truebit quickstart guide:
https://hub.docker.com/r/truebitprotocol/truebit-eth
You’ll have to use sudo when setting up and connecting to Docker. Start by pulling and installing Truebit OS. Then run the Linux incantation. This will create the truebit-docker files in your Home folder. Do the Clef initialization and set the auto-signing attest. Make at least two accounts, and make sure you back up your new accounts by going into truebit-geth, opening the keystore file, and copying the files there. These are your private keys, so keep them safe.
You might be wondering at this point how we’re going to get our newly synced node into the container.
When you perform the Linux incantation, you are manually creating the directories that will be used by the container. It seems logical to me that you could enter the full path to existing directories and use those instead of creating new ones, but given my ignorance of how these commands work, I was concerned about accidentally overwriting or otherwise corrupting my chaindata. So I didn’t touch those.
I also tried editing mainnet.sh, which is the config file that the container uses to launch Geth, Clef, and IPFS. There you can edit the geth.providerURL to point at the geth.ipc file in your local ethereum directory, but I didn’t have any luck doing this.
I’m sure there are better, less caveman ways to shove a local full node into a container, but here’s how I eventually did it: by copying the entire local geth folder into docker-geth.
This may not sound difficult on its face, but on Ubuntu, you will need to go into your truebit-docker folder and unlock every single file you intend to replace. To do this, go to Home, open the truebit-docker folder, and you will notice the name of the folder in the upper left of the window. Click this to open a drop-down menu, then click Open in Terminal. For each file you need to unlock, you will need to enter:
$sudo chown -R < username > < filename >
Start by unlocking the docker-geth folder. Then go inside, and systematically unlock every locked file in there, in every folder. You can delete your les-client and light client data folders if they’re in there.
Once this is done, go find your local geth folder in your local ethereum directory. Ubuntu is tricksome and hides this from you, but you can get at it by opening Files, clicking + Other Locations in the lower left, opening Computer, and searching for ethereum. Press CTRL+H to reveal hidden files. In the .ethereum folder you will find your local geth folder.
Again, make sure that you’ve shut down and exited your new node. Trying to copy geth while it is running can corrupt your chaindata.
Right click the geth folder, select Copy to…, and copy it into docker-geth. You’ll be asked to merge it with the geth in the container files. Go for it. If you missed unlocking any files, the system will let you know. Make note of which ones you missed, and make sure you replace them too once you’ve gone back and unlocked them.
Now that you’ve copied over your local geth directory into the container files, there’s one more step to take before you can connect to the network.
Open a new terminal window, and launch a second instance of the container with:
sudo docker exec -it truebit /bin/bash
Then type:
nano -w /mainnet.sh
The nano command allows you to edit a config file within the Terminal. You’ll now see the lines of code that constitute mainnet.sh. Use the arrow keys to navigate down to the line that begins GETH=$(echo …., and edit the parenthetical so that it reads:
GETH=$(echo ‘geth console --nousb --cache 10204 --gpo.blocks 10 --gpo.percentile 90 --signer’ $CLEF)
This should configure Geth to launch in default (fast) mode, with a cache of 10 GB, with a gas price oracle that reads the last 10 blocks and determines a gas price based on the 90th percentile upper bound of gas prices in those blocks. You can then exit with CTRL-X, where you will be prompted to save; press y, then press Enter.
Please note that if you ever restart the container (say, by exiting it completely, then reinitializing it later), you will need to edit mainnet.sh all over again.
After exiting nano, you can check your edit with the command:
cat /mainnet.sh
If it looks good, now in one of your Terminal windows you can run:
bash /mainnet.sh
This will bring up the split screen that you should be familiar with after running your test node on Goerli. Enter your Clef password, and the node should start looking for peers and syncing. If the Geth half of the window vanishes, or if after a moment you see the interminable flood of imported states into a node rebuilding the chain from scratch, something went wrong and you need to check either your truebit-geth directory or mainnet.sh.
If, however, you see the familiar stream of ‘Imported new chain segment’, ‘Unindexed transactions’, and ‘Deep froze chain segment’, congratulations! You now have a full node running inside the Truebit container.
In the second Terminal window, you can now run:
cd / truebit-eth
./truebit-os
To bring up Truebit OS. Here use:
accounts
To check that the accounts you made with Clef are there. You should also set:
gas set -v geth
To use your full node’s gas price oracle.
One final note: if you need to shutdown your container node, you will need to kill the process.
First, in the Terminal window where you’re running Truebit OS:
exit
out of Truebit OS. Then type:
ps aux | grep geth
Which will bring up a list of processes running in the container. In the readout you will see several iterations of “root” followed by a long space and a sequence of numbers. The one you want should be the third one down, where you’ll recognize the line you edited in mainnet.sh. After root and the long space, the first number in the sequence is the < pid >.
Now type
kill -INT < pid >
which will end the container Geth session. Allow the process to fully exit the blockchain, then:
exit
out of the container.