Improve protocol versioning between tier1 and tier2 #468
Description
In the substreams 1.6.0 release notes (https://github.com/streamingfast/substreams/releases/tag/v1.6.0), it says "Upgrading to v1.6.0 will require changing the tier1 and tier2 versions concurrently, as the internal protocol has changed."
If you don't upgrade them both together the tier1 gets a fatal message from the tier2 and then sends fatal (do not retry) error back to the client.
The block stream encountered a substreams fatal error and will not retry: rpc error: code = InvalidArgument desc = validate request: validate tier2 request: stop block 0 should be higher than start block 0
This is an enhancement to have tier1 and tier2 actually negotiate a proper version to avoid this from happening.
Tier1 should sent tier2 a version number in the connect message. If tier2 doesn't understand that version, it should send a non-fatal error back to tier1 (like the "tier2 is too busy now" message). Tier1 will try other teir2 nodes in the cluster presumably.
This is to prevent a single mis-configured tier2 node causing the whole substream to terminate.
Prometheus counter on tier1 can record how many times it gets bad version from tier2.