diff --git a/cometbft/v0.38/docs/README.mdx b/cometbft/v0.38/docs/README.mdx new file mode 100644 index 00000000..174c9e87 --- /dev/null +++ b/cometbft/v0.38/docs/README.mdx @@ -0,0 +1,44 @@ +--- +title: CometBFT Documentation +description: CometBFT is a blockchain application platform. +footer: + newsletter: false +--- +{/* trigger rebuild */} + +# CometBFT + +Welcome to the CometBFT documentation! + +CometBFT is a blockchain application platform; it provides the equivalent +of a web-server, database, and supporting libraries for blockchain applications +written in any programming language. Like a web-server serving web applications, +CometBFT serves blockchain applications. + +More formally, CometBFT performs Byzantine Fault Tolerant (BFT) +State Machine Replication (SMR) for arbitrary deterministic, finite state machines. +For more background, see [What is CometBFT?](./introduction/README.md#what-is-cometbft). + +To get started quickly with an example application, see the [quick start guide](guides/quick-start.md). + +To learn about application development on CometBFT, see the [Application Blockchain Interface](https://github.com/cometbft/cometbft/tree/v0.38.x/spec/abci). + +For more details on using CometBFT, see the respective documentation for +[CometBFT internals](core/), [benchmarking and monitoring](tools/), and [network deployments](networks/). + +## Contribute + +To recommend a change to the documentation, please submit a PR. Each major +release's documentation is housed on the corresponding release branch, e.g. for +the v0.34 release series, the documentation is housed on the `v0.34.x` branch. + +When submitting changes that affect all releases, please start by submitting a +PR to the docs on `main` - this will be backported to the relevant release +branches. If a change is exclusively relevant to a specific release, please +target that release branch with your PR. + +Changes to the documentation will be reviewed by the team and, if accepted and +merged, published to (https://docs.cometbft.com) for the respective version(s). + +The build process for the documentation is housed in the [CometBFT documentation +repository](https://github.com/cometbft/cometbft-docs). diff --git a/cometbft/v0.38/docs/app-dev/Application-Architecture-Guide.mdx b/cometbft/v0.38/docs/app-dev/Application-Architecture-Guide.mdx new file mode 100644 index 00000000..cf0cc53b --- /dev/null +++ b/cometbft/v0.38/docs/app-dev/Application-Architecture-Guide.mdx @@ -0,0 +1,55 @@ +--- +order: 4 +--- + +# Application Architecture Guide + +Here we provide a brief guide on the recommended architecture of a +CometBFT blockchain application. + +We distinguish here between two forms of "application". The first is the +end-user application, like a desktop-based wallet app that a user downloads, +which is where the user actually interacts with the system. The other is the +ABCI application, which is the logic that actually runs on the blockchain. +Transactions sent by an end-user application are ultimately processed by the ABCI +application after being committed by CometBFT. + +The end-user application communicates with a REST API exposed by the application. +The application runs CometBFT nodes and verifies CometBFT light-client proofs +through the CometBFT RPC. The CometBFT process communicates with +a local ABCI application, where the user query or transaction is actually +processed. + +The ABCI application must be a deterministic result of the CometBFT +consensus - any external influence on the application state that didn't +come through CometBFT could cause a consensus failure. Thus _nothing_ +should communicate with the ABCI application except CometBFT via ABCI. + +If the ABCI application is written in Go, it can be compiled into the +CometBFT binary. Otherwise, it should use a unix socket to communicate +with CometBFT. If it's necessary to use TCP, extra care must be taken +to encrypt and authenticate the connection. + +All reads from the ABCI application happen through the CometBFT `/abci_query` +endpoint. All writes to the ABCI application happen through the CometBFT +`/broadcast_tx_*` endpoints. + +The Light-Client Daemon is what provides light clients (end users) with +nearly all the security of a full node. It formats and broadcasts +transactions, and verifies proofs of queries and transaction results. +Note that it need not be a daemon - the Light-Client logic could instead +be implemented in the same process as the end-user application. + +Note for those ABCI applications with weaker security requirements, the +functionality of the Light-Client Daemon can be moved into the ABCI +application process itself. That said, exposing the ABCI application process +to anything besides CometBFT over ABCI requires extreme caution, as +all transactions, and possibly all queries, should still pass through +CometBFT. + +See the following for more extensive documentation: + +- [Interchain Standard for the Light-Client REST API](https://github.com/cosmos/cosmos-sdk/pull/1617) (legacy/deprecated) +- [CometBFT RPC Docs](../rpc) +- [CometBFT in Production](../core/running-in-production.md) +- [ABCI spec](../spec/abci) diff --git a/cometbft/v0.38/docs/app-dev/Getting-Started.mdx b/cometbft/v0.38/docs/app-dev/Getting-Started.mdx new file mode 100644 index 00000000..8a23cad3 --- /dev/null +++ b/cometbft/v0.38/docs/app-dev/Getting-Started.mdx @@ -0,0 +1,199 @@ +--- +order: 2 +--- + +# Getting Started + +## First CometBFT App + +As a general purpose blockchain engine, CometBFT is agnostic to the +application you want to run. So, to run a complete blockchain that does +something useful, you must start two programs: one is CometBFT, +the other is your application, which can be written in any programming +language. + +CometBFT handles all the p2p and consensus logic, and just forwards transactions to the +application when they need to be validated, or when they're ready to be +executed and committed. + +In this guide, we show you some examples of how to run an application +using CometBFT. + +### Install + +The first apps we will work with are written in Go. To install them, you +need to [install Go](https://golang.org/doc/install), put +`$GOPATH/bin` in your `$PATH` and enable go modules. If you use `bash`, +follow these instructions: + +```bash +echo export GOPATH=\"\$HOME/go\" >> ~/.bash_profile +echo export PATH=\"\$PATH:\$GOPATH/bin\" >> ~/.bash_profile +``` + +Then run + +```bash +go get github.com/cometbft/cometbft +cd $GOPATH/src/github.com/cometbft/cometbft +make install_abci +``` + +Now you should have the `abci-cli` installed; run `abci-cli` to see the list of commands: + +``` +Usage: + abci-cli [command] + +Available Commands: + batch run a batch of abci commands against an application + check_tx validate a transaction + commit commit the application state and return the Merkle root hash + completion Generate the autocompletion script for the specified shell + console start an interactive ABCI console for multiple commands + echo have the application echo a message + finalize_block deliver a block of transactions to the application + help Help about any command + info get some info about the application + kvstore ABCI demo example + prepare_proposal prepare proposal + process_proposal process proposal + query query the application state + test run integration tests + version print ABCI console version + +Flags: + --abci string either socket or grpc (default "socket") + --address string address of application socket (default "tcp://0.0.0.0:26658") + -h, --help help for abci-cli + --log_level string set the logger level (default "debug") + -v, --verbose print the command and results as if it were a console session + +Use "abci-cli [command] --help" for more information about a command. +``` + +You'll notice the `kvstore` command, an example application written in Go. + +Now, let's run an app! + +## KVStore - A First Example + +The kvstore app is a [Merkle +tree](https://en.wikipedia.org/wiki/Merkle_tree) that just stores all +transactions. If the transaction contains an `=`, e.g. `key=value`, then +the `value` is stored under the `key` in the Merkle tree. Otherwise, the +full transaction bytes are stored as the key and the value. + +Let's start a kvstore application. + +```sh +abci-cli kvstore +``` + +In another terminal, we can start CometBFT. You should already have the +CometBFT binary installed. If not, follow the steps from +[here](../guides/install.md). If you have never run CometBFT +before, use: + +```sh +cometbft init +cometbft node +``` + +If you have used CometBFT, you may want to reset the data for a new +blockchain by running `cometbft unsafe-reset-all`. Then you can run +`cometbft node` to start CometBFT, and connect to the app. For more +details, see [the guide on using CometBFT](../core/using-cometbft.md). + +You should see CometBFT making blocks! We can get the status of our +CometBFT node as follows: + +```sh +curl -s localhost:26657/status +``` + +The `-s` just silences `curl`. For nicer output, pipe the result into a +tool like [jq](https://stedolan.github.io/jq/) or `json_pp`. + +Now let's send some transactions to the kvstore. + +```sh +curl -s 'localhost:26657/broadcast_tx_commit?tx="abcd"' +``` + +Note the single quote (`'`) around the url, which ensures that the +double quotes (`"`) are not escaped by bash. This command sent a +transaction with bytes `abcd`, so `abcd` will be stored as both the key +and the value in the Merkle tree. The response should look something +like: + +```json +{ + "jsonrpc": "2.0", + "id": "", + "result": { + "check_tx": {}, + "deliver_tx": { + "tags": [ + { + "key": "YXBwLmNyZWF0b3I=", + "value": "amFl" + }, + { + "key": "YXBwLmtleQ==", + "value": "YWJjZA==" + } + ] + }, + "hash": "9DF66553F98DE3C26E3C3317A3E4CED54F714E39", + "height": 14 + } +} +``` + +We can confirm that our transaction worked and the value got stored by +querying the app: + +```sh +curl -s 'localhost:26657/abci_query?data="abcd"' +``` + +The result should look like: + +```json +{ + "jsonrpc": "2.0", + "id": "", + "result": { + "response": { + "log": "exists", + "index": "-1", + "key": "YWJjZA==", + "value": "YWJjZA==" + } + } +} +``` + +Note the `value` in the result (`YWJjZA==`); this is the base64-encoding +of the ASCII of `abcd`. You can verify this in a python 2 shell by +running `"YWJjZA==".decode('base64')` or in python 3 shell by running +`import codecs; codecs.decode(b"YWJjZA==", 'base64').decode('ascii')`. +Stay tuned for a future release that [makes this output more +human-readable](https://github.com/tendermint/tendermint/issues/1794). + +Now let's try setting a different key and value: + +```sh +curl -s 'localhost:26657/broadcast_tx_commit?tx="name=satoshi"' +``` + +Now if we query for `name`, we should get `satoshi`, or `c2F0b3NoaQ==` +in base64: + +```sh +curl -s 'localhost:26657/abci_query?data="name"' +``` + +Try some other transactions and queries to make sure everything is +working! diff --git a/cometbft/v0.38/docs/app-dev/Indexing-Transactions.mdx b/cometbft/v0.38/docs/app-dev/Indexing-Transactions.mdx new file mode 100644 index 00000000..f20f173e --- /dev/null +++ b/cometbft/v0.38/docs/app-dev/Indexing-Transactions.mdx @@ -0,0 +1,305 @@ +--- +order: 5 +--- + +# Indexing Transactions + +CometBFT allows you to index transactions and blocks and later query or +subscribe to their results. Transactions are indexed by `ResponseFinalizeBlock.tx_results.events` and +blocks are indexed by `ResponseFinalizeBlock.events`. However, transactions +are also indexed by a primary key which includes the transaction hash and maps +to and stores the corresponding transaction results. Blocks are indexed by a primary key +which includes the block height and maps to and stores the block height, i.e. +the block itself is never stored. + +Each event contains a type and a list of attributes, which are key-value pairs +denoting something about what happened during the method's execution. For more +details on `Events`, see the [ABCI][abci-events] documentation. + +An `Event` has a composite key associated with it. A `compositeKey` is +constructed by its type and key separated by a dot. + +For example: + +```json +"jack": [ + "account.number": 100 +] +``` + +would be equal to the composite key of `jack.account.number`. + +By default, CometBFT will index all transactions by their respective hashes +and height and blocks by their height. + +CometBFT allows for different events within the same height to have +equal attributes. + +## Configuration + +Operators can configure indexing via the `[tx_index]` section. The `indexer` +field takes a series of supported indexers. If `null` is included, indexing will +be turned off regardless of other values provided. + +```toml +[tx-index] + +# The backend database to back the indexer. +# If indexer is "null", no indexer service will be used. +# +# The application will set which txs to index. In some cases a node operator will be able +# to decide which txs to index based on configuration set in the application. +# +# Options: +# 1) "null" +# 2) "kv" (default) - the simplest possible indexer, backed by key-value storage (defaults to levelDB; see DBBackend). +# - When "kv" is chosen "tx.height" and "tx.hash" will always be indexed. +# 3) "psql" - the indexer services backed by PostgreSQL. +# indexer = "kv" +``` + +### Supported Indexers + +#### KV + +The `kv` indexer type is an embedded key-value store supported by the main +underlying CometBFT database. Using the `kv` indexer type allows you to query +for block and transaction events directly against CometBFT's RPC. However, the +query syntax is limited and so this indexer type might be deprecated or removed +entirely in the future. + +**Implementation and data layout** + +The kv indexer stores each attribute of an event individually, by creating a composite key +with + +- event type, +- attribute key, +- attribute value, +- event generator (e.g. `FinalizeBlock`) +- the height, and +- event counter. + For example the following events: + +``` +Type: "transfer", + Attributes: []abci.EventAttribute{ + {Key: "sender", Value: "Bob", Index: true}, + {Key: "recipient", Value: "Alice", Index: true}, + {Key: "balance", Value: "100", Index: true}, + {Key: "note", Value: "nothing", Index: true}, + }, + +``` + +``` +Type: "transfer", + Attributes: []abci.EventAttribute{ + {Key: "sender", Value: "Tom", Index: true}, + {Key: "recipient", Value: "Alice", Index: true}, + {Key: "balance", Value: "200", Index: true}, + {Key: "note", Value: "nothing", Index: true}, + }, +``` + +will be represented as follows in the store, assuming these events result from the `FinalizeBlock` call for height 1: + +``` +Key value +---- event1 ------ +transferSenderBobFinalizeBlock11 1 +transferRecipientAliceFinalizeBlock11 1 +transferBalance100FinalizeBlock11 1 +transferNodeNothingFinalizeBlock11 1 +---- event2 ------ +transferSenderTomFinalizeBlock12 1 +transferRecepientAliceFinalizeBlock12 1 +transferBalance200FinalizeBlock12 1 +transferNodeNothingFinalizeBlock12 1 + +``` + +The event number is a local variable kept by the indexer and incremented when a new event is processed. +It is an `int64` variable and has no other semantics besides being used to associate attributes belonging to the same events within a height. +This variable is not atomically incremented as event indexing is deterministic. **Should this ever change**, the event id generation +will be broken. + +#### PostgreSQL + +The `psql` indexer type allows an operator to enable block and transaction event +indexing by proxying it to an external PostgreSQL instance allowing for the events +to be stored in relational models. Since the events are stored in a RDBMS, operators +can leverage SQL to perform a series of rich and complex queries that are not +supported by the `kv` indexer type. Since operators can leverage SQL directly, +searching is not enabled for the `psql` indexer type via CometBFT's RPC -- any +such query will fail. + +Note, the SQL schema is stored in `state/indexer/sink/psql/schema.sql` and operators +must explicitly create the relations prior to starting CometBFT and enabling +the `psql` indexer type. + +Example: + +```shell +psql ... -f state/indexer/sink/psql/schema.sql +``` + +## Default Indexes + +The CometBFT tx and block event indexer indexes a few select reserved events +by default. + +### Transactions + +The following indexes are indexed by default: + +- `tx.height` +- `tx.hash` + +### Blocks + +The following indexes are indexed by default: + +- `block.height` + +## Adding Events + +Applications are free to define which events to index. CometBFT does not +expose functionality to define which events to index and which to ignore. In +your application's `FinalizeBlock` method, add the `Events` field with pairs of +UTF-8 encoded strings (e.g. "transfer.sender": "Bob", "transfer.recipient": +"Alice", "transfer.balance": "100"). + +Example: + +```go +func (app *Application) FinalizeBlock(_ context.Context, req *types.RequestFinalizeBlock) (*types.ResponseFinalizeBlock, error) { + + //... + tx_results[0] := &types.ExecTxResult{ + Code: CodeTypeOK, + // With every transaction we can emit a series of events. To make it simple, we just emit the same events. + Events: []types.Event{ + { + Type: "app", + Attributes: []types.EventAttribute{ + {Key: "creator", Value: "Cosmoshi Netowoko", Index: true}, + {Key: "key", Value: key, Index: true}, + {Key: "index_key", Value: "index is working", Index: true}, + {Key: "noindex_key", Value: "index is working", Index: false}, + }, + }, + { + Type: "app", + Attributes: []types.EventAttribute{ + {Key: "creator", Value: "Cosmoshi", Index: true}, + {Key: "key", Value: value, Index: true}, + {Key: "index_key", Value: "index is working", Index: true}, + {Key: "noindex_key", Value: "index is working", Index: false}, + }, + }, + }, + } + + block_events = []types.Event{ + { + Type: "loan", + Attributes: []types.EventAttribute{ + { Key: "account_no", Value: "1", Index: true}, + { Key: "amount", Value: "200", Index: true }, + }, + }, + { + Type: "loan", + Attributes: []types.EventAttribute{ + { Key: "account_no", Value: "2", Index: true }, + { Key: "amount", Value: "300", Index: true}, + }, + }, + } + return &types.ResponseFinalizeBlock{TxResults: tx_results, Events: block_events} +} +``` + +If the indexer is not `null`, the transaction will be indexed. Each event is +indexed using a composite key in the form of `{eventType}.{eventAttribute}={eventValue}`, +e.g. `transfer.sender=bob`. + +## Querying Transactions Events + +You can query for a paginated set of transaction by their events by calling the +`/tx_search` RPC endpoint: + +```bash +curl "localhost:26657/tx_search?query=\"message.sender='cosmos1...'\"&prove=true" +``` + +Check out [API docs](https://docs.cometbft.com/v0.38/rpc/#/Info/tx_search) +for more information on query syntax and other options. + +## Subscribing to Transactions + +Clients can subscribe to transactions with the given tags via WebSocket by providing +a query to `/subscribe` RPC endpoint. + +```json +{ + "jsonrpc": "2.0", + "method": "subscribe", + "id": "0", + "params": { + "query": "message.sender='cosmos1...'" + } +} +``` + +Check out [API docs](https://docs.cometbft.com/v0.38/rpc/#subscribe) for more information +on query syntax and other options. + +## Querying Block Events + +You can query for a paginated set of blocks by their events by calling the +`/block_search` RPC endpoint: + +```bash +curl "localhost:26657/block_search?query=\"block.height > 10\"" +``` + + +Storing the event sequence was introduced in CometBFT 0.34.26. Before that, up +until Tendermint Core 0.34.26, the event sequence was not stored in the kvstore +and events were stored only by height. That means that queries returned blocks +and transactions whose event attributes match within the height but can match +across different events on that height. + +This behavior was fixed with CometBFT 0.34.26+. However, if the data was +indexed with earlier versions of Tendermint Core and not re-indexed, that data +will be queried as if all the attributes within a height occurred within the +same event. + +## Event attribute value types + +Users can use anything as an event value. However, if the event attribute value +is a number, the following needs to be taken into account: + +- Negative numbers will not be properly retrieved when querying the indexer. +- Event values are converted to big floats (from the `big/math` package). The + precision of the floating point number is set to the bit length of the + integer it is supposed to represent, so that there is no loss of information + due to insufficient precision. This was not present before CometBFT v0.38.x + and all float values were ignored. +- As of CometBFT v0.38.x, queries can contain floating point numbers as well. +- Note that comparing to floats can be imprecise with a high number of decimals. + +## Event type and attribute key format + +An event type/attribute key is a string that can contain any Unicode letter or +digit, as well as the following characters: `.` (dot), `-` (dash), `_` +(underscore). The event type/attribute key must not start with `-` (dash) or +`.` (dot). + +``` +^[\w]+[\.-\w]?$ +``` + +[abci-events]: ../spec/abci/abci++_basic_concepts.md#events diff --git a/cometbft/v0.38/docs/app-dev/Using-ABCI-CLI.mdx b/cometbft/v0.38/docs/app-dev/Using-ABCI-CLI.mdx new file mode 100644 index 00000000..829a9a6c --- /dev/null +++ b/cometbft/v0.38/docs/app-dev/Using-ABCI-CLI.mdx @@ -0,0 +1,213 @@ +--- +order: 3 +--- + +# Using ABCI-CLI + +To facilitate testing and debugging of ABCI servers and simple apps, we +built a CLI, the `abci-cli`, for sending ABCI messages from the command +line. + +## Install + +Make sure you [have Go installed](https://golang.org/doc/install). + +Next, install the `abci-cli` tool and example applications: + +```sh +git clone https://github.com/cometbft/cometbft.git +cd cometbft +make install_abci +``` + +Now run `abci-cli` to see the list of commands: + +```sh +Usage: + abci-cli [command] + +Available Commands: + batch run a batch of abci commands against an application + check_tx validate a transaction + commit commit the application state and return the Merkle root hash + completion Generate the autocompletion script for the specified shell + console start an interactive ABCI console for multiple commands + echo have the application echo a message + finalize_block deliver a block of transactions to the application + help Help about any command + info get some info about the application + kvstore ABCI demo example + prepare_proposal prepare proposal + process_proposal process proposal + query query the application state + test run integration tests + version print ABCI console version + +Flags: + --abci string either socket or grpc (default "socket") + --address string address of application socket (default "tcp://0.0.0.0:26658") + -h, --help help for abci-cli + --log_level string set the logger level (default "debug") + -v, --verbose print the command and results as if it were a console session + +Use "abci-cli [command] --help" for more information about a command. +``` + +## KVStore - First Example + +The `abci-cli` tool lets us send ABCI messages to our application, to +help build and debug them. + +The most important messages are `deliver_tx`, `check_tx`, and `commit`, +but there are others for convenience, configuration, and information +purposes. + +We'll start a kvstore application, which was installed at the same time as +`abci-cli` above. The kvstore just stores transactions in a Merkle tree. Its +code can be found +[here](https://github.com/cometbft/cometbft/blob/v0.38.x/abci/example/kvstore/kvstore.go). + +Start the application by running: + +```sh +abci-cli kvstore +``` + +And in another terminal, run + +```sh +abci-cli echo hello +abci-cli info +``` + +You'll see something like: + +```sh +-> data: hello +-> data.hex: 68656C6C6F +``` + +and: + +```sh +-> data: {"size":0} +-> data.hex: 7B2273697A65223A307D +``` + +An ABCI application must provide two things: + +- a socket server +- a handler for ABCI messages + +When we run the `abci-cli` tool we open a new connection to the +application's socket server, send the given ABCI message, and wait for a +response. + +The server may be generic for a particular language, and we provide a +[reference implementation in +Golang](https://github.com/cometbft/cometbft/tree/v0.38.x/abci/server). See the +[list of other ABCI implementations](https://github.com/tendermint/awesome#ecosystem) for servers in +other languages. + +The handler is specific to the application, and may be arbitrary, so +long as it is deterministic and conforms to the ABCI interface +specification. + +So when we run `abci-cli info`, we open a new connection to the ABCI +server, which calls the `Info()` method on the application, which tells +us the number of transactions in our Merkle tree. + +Now, since every command opens a new connection, we provide the +`abci-cli console` and `abci-cli batch` commands, to allow multiple ABCI +messages to be sent over a single connection. + +Running `abci-cli console` should drop you in an interactive console for +speaking ABCI messages to your application. + +Try running these commands: + +```sh +> echo hello +-> code: OK +-> data: hello +-> data.hex: 0x68656C6C6F + +> info +-> code: OK +-> data: {"size":0} +-> data.hex: 0x7B2273697A65223A307D + +> prepare_proposal "abc=123" +-> code: OK +-> log: Succeeded. Tx: abc=123 + +> process_proposal "abc==456" +-> code: OK +-> status: REJECT + +> process_proposal "abc=123" +-> code: OK +-> status: ACCEPT + +> finalize_block "abc=123" +-> code: OK +-> code: OK +-> data.hex: 0x0200000000000000 + +> commit +-> code: OK + +> info +-> code: OK +-> data: {"size":1} +-> data.hex: 0x7B2273697A65223A317D + +> query "abc" +-> code: OK +-> log: exists +-> height: 0 +-> key: abc +-> key.hex: 616263 +-> value: 123 +-> value.hex: 313233 + +> finalize_block "def=xyz" "ghi=123" +-> code: OK +-> code: OK +-> code: OK +-> data.hex: 0x0600000000000000 + +> commit +-> code: OK + +> query "def" +-> code: OK +-> log: exists +-> height: 0 +-> key: def +-> key.hex: 646566 +-> value: xyz +-> value.hex: 78797A +``` + +Note that if we do `finalize_block "abc" ...` it will store `(abc, abc)`, but if +we do `finalize_block "abc=efg" ...` it will store `(abc, efg)`. + +You could put the commands in a file and run +`abci-cli --verbose batch < myfile`. + + +Note that the `abci-cli` is designed strictly for testing and debugging. In a real +deployment, the role of sending messages is taken by CometBFT, which +connects to the app using four separate connections, each with its own +pattern of messages. + +For examples of running an ABCI app with CometBFT, see the +[getting started guide](./getting-started.md). + +## Bounties + +Want to write an app in your favorite language?! We'd be happy +to add you to our [ecosystem](https://github.com/tendermint/awesome#ecosystem)! +See [funding](https://github.com/interchainio/funding) opportunities from the +[Interchain Foundation](https://interchain.io) for implementations in new languages and more. diff --git a/cometbft/v0.38/docs/architecture/README.md b/cometbft/v0.38/docs/architecture/README.md new file mode 100644 index 00000000..761c6827 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/README.md @@ -0,0 +1,54 @@ +--- +order: 1 +parent: + order: false +--- + +# Architecture Decision Records (ADR) + +This is a location to record all high-level architecture decisions in the +CometBFT project. + +You can read more about the ADR concept in this +[blog post](https://product.reverb.com/documenting-architecture-decisions-the-reverb-way-a3563bb24bd0#.78xhdix6t). + +An ADR should, with a strong focus on the impact on _users_ of the system, +provide: + +- Context on the relevant goals and the current state +- Proposed changes to achieve the goals +- Summary of pros and cons +- References +- Changelog + +To create a new ADR, please use the [ADR template](./adr-template.md). + +Note the distinction between an ADR and a spec. An ADR provides the context, +intuition, reasoning, and justification for a change in architecture, or for the +architecture of something new. A spec is more compressed and streamlined +summary of everything as it stands today. + +If recorded decisions turned out to be lacking, convene a discussion, record the +new decisions here, and then modify the code to match. + +Note the context/background should be written in the present tense. + +## Table of Contents + +The following ADRs are exclusively relevant to CometBFT. For historical ADRs +relevant to Tendermint Core as well, please see [this list](./tendermint-core/). +To distinguish CometBFT ADRs from historical ones from Tendermint Core, we start +numbering our ADRs from 100 onwards. + +### Proposed + +### Accepted + +- [ADR-111: `nop` Mempool](./adr-111-nop-mempool.md) + +### Implemented + +### Deprecated + +### Rejected + diff --git a/cometbft/v0.38/docs/architecture/adr-111-nop-mempool.md b/cometbft/v0.38/docs/architecture/adr-111-nop-mempool.md new file mode 100644 index 00000000..234cd5b9 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/adr-111-nop-mempool.md @@ -0,0 +1,324 @@ +# ADR 111: `nop` Mempool + +## Changelog + +- 2023-11-07: First version (@sergio-mena) +- 2023-11-15: Addressed PR comments (@sergio-mena) +- 2023-11-17: Renamed `nil` to `nop` (@melekes) +- 2023-11-20: Mentioned that the app could reuse p2p network in the future (@melekes) +- 2023-11-22: Adapt ADR to implementation (@melekes) + +## Status + +Accepted + +[Tracking issue](https://github.com/cometbft/cometbft/issues/1666) + +## Context + +### Summary + +The current mempool built into CometBFT implements a robust yet somewhat inefficient transaction gossip mechanism. +While the CometBFT team is currently working on more efficient general-purpose transaction gossiping mechanisms, +some users have expressed their desire to manage both the mempool and the transaction dissemination mechanism +outside CometBFT (typically at the application level). + +This ADR proposes a fairly simple way for CometBFT to fulfill this use case without moving away from our current architecture. + +### In the Beginning... + +It is well understood that a dissemination mechanism +(sometimes using _Reliable Broadcast_ [\[HT94\]][HT94] but not necessarily), +is needed in a distributed system implementing State-Machine Replication (SMR). +This is also the case in blockchains. +Early designs such as Bitcoin or Ethereum include an _internal_ component, +responsible for dissemination, called mempool. +Tendermint Core chose to follow the same design given the success +of those early blockchains and, since inception, Tendermint Core and later CometBFT have featured a mempool as an internal piece of its architecture. + + +However, the design of ABCI clearly dividing the application logic (i.e., the appchain) +and the consensus logic that provides SMR semantics to the app is a unique innovation in Cosmos +that sets it apart from Bitcoin, Ethereum, and many others. +This clear separation of concerns entailed many consequences, mostly positive: +it allows CometBFT to be used underneath (currently) tens of different appchains in production +in the Cosmos ecosystem and elsewhere. +But there are other implications for having an internal mempool +in CometBFT: the interaction between the mempool, the application, and the network +becomes more indirect, and thus more complex and hard to understand and operate. + +### ABCI++ Improvements and Remaining Shortcomings + +Before the release of ABCI++, `CheckTx` was the main mechanism the app had at its disposal to influence +what transactions made it to the mempool, and very indirectly what transactions got ultimately proposed in a block. +Since ABCI 1.0 (the first part of ABCI++, shipped in `v0.37.x`), the application has +a more direct say in what is proposed through `PrepareProposal` and `ProcessProposal`. + +This has greatly improved the ability for appchains to influence the contents of the proposed block. +Further, ABCI++ has enabled many new use cases for appchains. However some issues remain with +the current model: + +* We are using the same P2P network for disseminating transactions and consensus-related messages. +* Many mempool parameters are configured on a per-node basis by node operators, + allowing the possibility of inconsistent mempool configuration across the network + with potentially serious scalability effects + (even causing unacceptable performance degradation in some extreme cases). +* The current mempool implementation uses a basic (robust but sub-optimal) flood algorithm + * the CometBFT team is working on improving it as one of our current priorities, + but any improvement we come up with must address the needs of a vast spectrum of applications, + as well as be heavily scaled-tested in various scenarios + (in an attempt to cover the applications' wide spectrum) + * a mempool designed specifically for one particular application + would reduce the search space as its designers can devise it with just their application's + needs in mind. +* The interaction with the application is still somewhat convoluted: + * the application has to decide what logic to implement in `CheckTx`, + what to do with the transaction list coming in `RequestPrepareProposal`, + whether it wants to maintain an app-side mempool (more on this below), and whether or not + to combine the transactions in the app-side mempool with those coming in `RequestPrepareProposal` + * all those combinations are hard to fully understand, as the semantics and guarantees are + often not clear + * when using exclusively an app-mempool (the approach taken in the Cosmos SDK `v0.47.x`) + for populating proposed blocks, with the aim of simplifying the app developers' life, + we still have a suboptimal model where we need to continue using CometBFT's mempool + in order to disseminate the transactions. So, we end up using twice as much memory, + as in-transit transactions need to be kept in both mempools. + +The approach presented in this ADR builds on the app-mempool design released in `v0.47.x` +of the Cosmos SDK, +and briefly discussed in the last bullet point above (see [SDK app-mempool][sdk-app-mempool] for further details of this model). + +In the app-mempool design in Cosmos SDK `v0.47.x` +an unconfirmed transaction must be both in CometBFT's mempool for dissemination and +in the app's mempool so the application can decide how to manage the mempool. +There is no doubt that this approach has numerous advantages. However, it also has some implications that need to be considered: + +* Having every transaction both in CometBFT and in the application is suboptimal in terms of memory. + Additionally, the app developer has to be careful + that the contents of both mempools do not diverge over time + (hence the crucial role `re-CheckTx` plays post-ABCI++). +* The main reason for a transaction needing to be in CometBFT's mempool is + because the design in Cosmos SDK `v0.47.x` does not consider an application + that has its own means of disseminating transactions. + It reuses the peer to peer network underneath CometBFT reactors. +* There is no point in having transactions in CometBFT's mempool if an application implements an ad-hoc design for disseminating transactions. + +This proposal targets this kind of applications: +those that have an ad-hoc mechanism for transaction dissemination that better meets the application requirements. + +The ABCI application could reuse the P2P network once this is exposed via ABCI. +But this will take some time as it needs to be implemented, and has a dependency +on bi-directional ABCI, which is also quite substantial. See +[1](https://github.com/cometbft/cometbft/discussions/1112) and +[2](https://github.com/cometbft/cometbft/discussions/494) discussions. + +We propose to introduce a `nop` (short for no operation) mempool which will effectively act as a stubbed object +internally: + +* it will reject any transaction being locally submitted or gossipped by a peer +* when a _reap_ (as it is currently called) is executed in the mempool, an empty answer will always be returned +* the application running on the proposer validator will add transactions it received + using the appchains's own mechanism via `PrepareProposal`. + +## Alternative Approaches + +These are the alternatives known to date: + +1. Keep the current model. Useful for basic apps, but clearly suboptimal for applications + with their own mechanism to disseminate transactions and particular performance requirements. +2. Provide more efficient general-purpose mempool implementations. + This is an ongoing effort (e.g., [CAT mempool][cat-mempool]), but will take some time, and R&D effort, to come up with + advanced mechanisms -- likely highly configurable and thus complex -- which then will have to be thoroughly tested. +3. A similar approach to this one ([ADR110][adr-110]) whereby the application-specific + mechanism directly interacts with CometBFT via a newly defined gRPC interface. +4. Partially adopting this ADR. There are several possibilities: + * Use the current mempool, disable transaction broadcast in `config.toml`, and accept transactions from users via `BroadcastTX*` RPC methods. + Positive: avoids transaction gossiping; app can reuse the mempool existing in ComeBFT. + Negative: requires clients to know the validators' RPC endpoints (potential security issues). + * Transaction broadcast is disabled in `config.toml`, and have the application always reject transactions in `CheckTx`. + Positive: effectively disables the mempool; does not require modifications to Comet (may be used in `v0.37.x` and `v0.38.x`). + Negative: requires apps to disseminate txs themselves; the setup for this is less straightforward than this ADR's proposal. + +## Decision + +TBD + +## Detailed Design + +What this ADR proposes can already be achieved with an unmodified CometBFT since +`v0.37.x`, albeit with a complex, poor UX (see the last alternative in section +[Alternative Approaches](#alternative-approaches)). The core of this proposal +is to make some internal changes so it is clear an simple for app developers, +thus improving the UX. + +#### `nop` Mempool + +We propose a new mempool implementation, called `nop` Mempool, that effectively disables all mempool functionality +within CometBFT. +The `nop` Mempool implements the `Mempool` interface in a very simple manner: + +* `CheckTx(tx types.Tx) (*abcicli.ReqRes, error)`: returns `nil, ErrNotAllowed` +* `RemoveTxByKey(txKey types.TxKey) error`: returns `ErrNotAllowed` +* `ReapMaxBytesMaxGas(maxBytes, maxGas int64) types.Txs`: returns `nil` +* `ReapMaxTxs(max int) types.Txs`: returns `nil` +* `Lock()`: does nothing +* `Unlock()`: does nothing +* `Update(...) error`: returns `nil` +* `FlushAppConn() error`: returns `nil` +* `Flush()`: does nothing +* `TxsAvailable() <-chan struct{}`: returns `nil` +* `EnableTxsAvailable()`: does nothing +* `SetTxRemovedCallback(cb func(types.TxKey))`: does nothing +* `Size() int` returns 0 +* `SizeBytes() int64` returns 0 + +Upon startup, the `nop` mempool reactor will advertise no channels to the peer-to-peer layer. + +### Configuration + +We propose the following changes to the `config.toml` file: + +```toml +[mempool] +# The type of mempool for this CometBFT node to use. +# +# Valid types of mempools supported by CometBFT: +# - "flood" : clist mempool with flooding gossip protocol (default) +# - "nop" : nop-mempool (app has implemented an alternative tx dissemination mechanism) +type = "nop" +``` + +The config validation logic will be modified to add a new rule that rejects a configuration file +if all of these conditions are met: + +* the mempool is set to `nop` +* `create_empty_blocks`, in `consensus` section, is set to `false`. + +The reason for this extra validity rule is that the `nop`-mempool, as proposed here, +does not support the "do not create empty blocks" functionality. +Here are some considerations on this: + +* The "do not create empty blocks" functionality + * entangles the consensus and mempool reactors + * is hardly used in production by real appchains (to the best of CometBFT team's knowledge) + * its current implementation for the built-in mempool has undesired side-effects + * app hashes currently refer to the previous block, + * and thus it interferes with query provability. +* If needed in the future, this can be supported by extending ABCI, + but we will first need to see a real need for this before committing to changing ABCI + (which has other, higher-impact changes waiting to be prioritized). + +### RPC Calls + +There are no changes needed in the code dealing with RPC. Those RPC paths that call methods of the `Mempool` interface, +will simply be calling the new implementation. + +### Impacted Workflows + +* *Submitting a transaction*. Users are not to submit transactions via CometBFT's RPC. + `BroadcastTx*` RPC methods will fail with a reasonable error and the 501 status code. + The application running on a full node must offer an interface for users to submit new transactions. + It could also be a distinct node (or set of nodes) in the network. + These considerations are exclusively the application's concern in this approach. +* *Time to propose a block*. The consensus reactor will call `ReapMaxBytesMaxGas` which will return a `nil` slice. + `RequestPrepareProposal` will thus contain no transactions. +* *Consensus waiting for transactions to become available*. `TxsAvailable()` returns `nil`. + `cs.handleTxsAvailable()` won't ever be executed. + At any rate, a configuration with the `nop` mempool and `create_empty_blocks` set to `false` + will be rejected in the first place. +* *A new block is decided*. + * When `Update` is called, nothing is done (no decided transaction is removed). + * Locking and unlocking the mempool has no effect. +* *ABCI mempool's connection* + CometBFT will still open a "mempool" connection, even though it won't be used. + This is to avoid doing lots of breaking changes. + +### Impact on Current Release Plans + +The changes needed for this approach, are fairly simple, and the logic is clear. +This might allow us to even deliver it as part of CometBFT `v1` (our next release) +even without a noticeable impact on `v1`'s delivery schedule. + +The CometBFT team (learning from past dramatic events) usually takes a conservative approach +for backporting changes to release branches that have already undergone a full QA cycle +(and thus are in code-freeze mode). +For this reason, although the limited impact of these changes would limit the risks +of backporting to `v0.38.x` and `v0.37.x`, a careful risk/benefit evaluation will +have to be carried out. + +Backporting to `v0.34.x` does not make sense as this version predates the release of `ABCI 1.0`, +so using the `nop` mempool renders CometBFT's operation useless. + +### Config parameter _vs._ application-enforced parameter + +In the current proposal, the parameter selecting the mempool is in `config.toml`. +However, it is not a clear-cut decision. These are the alternatives we see: + +* *Mempool selected in `config.toml` (our current design)*. + This is the way the mempool has always been selected in Tendermint Core and CometBFT, + in those versions where there were more than one mempool to choose from. + As the configuration is in `config.toml`, it is up to the node operators to configure their + nodes consistently, via social consensus. However this cannot be guaranteed. + A network with an inconsistent choice of mempool at different nodes might + result in undesirable side effects, such as peers disconnecting from nodes + that sent them messages via the mempool channel. +* *Mempool selected as a network-wide parameter*. + A way to prevent any inconsistency when selecting the mempool is to move the configuration out of `config.toml` + and have it as a network-wide application-enforced parameter, implemented in the same way as Consensus Params. + The Cosmos community may not be ready for such a rigid, radical change, + even if it eliminates the risk of operators shooting themselves in the foot. + Hence we went currently favor the previous alternative. +* *Mempool selected as a network-wide parameter, but allowing override*. + A third option, half way between the previous two, is to have the mempool selection + as a network-wide parameter, but with a special value called _local-config_ that still + allows an appchain to decide to leave it up to operators to configure it in `config.toml`. + +Ultimately, the "config parameter _vs._ application-enforced parameter" discussion +is a more general one that is applicable to other parameters not related to mempool selection. +In that sense, it is out of the scope of this ADR. + +## Consequences + +### Positive + +- Applications can now find mempool mechanisms that fit better their particular needs: + - Ad-hoc ways to add, remove, merge, reorder, modify, prioritize transactions according + to application needs. + - A way to disseminate transactions (gossip-based or other) to get the submitted transactions + to proposers. The application developers can devise simpler, efficient mechanisms tailored + to their application. + - Back-pressure mechanisms to prevent malicious users from abusing the transaction + dissemination mechanism. +- In this approach, CometBFT's peer-to-peer layer is relieved from managing transaction gossip, freeing up its resources for other reactors such as consensus, evidence, block-sync, or state-sync. +- There is no risk for the operators of a network to provide inconsistent configurations + for some mempool-related parameters. Some of those misconfigurations are known to have caused + serious performance issues in CometBFT's peer to peer network. + Unless, of course, the application-defined transaction dissemination mechanism ends up + allowing similar configuration inconsistencies. +- The interaction between the application and CometBFT at `PrepareProposal` time + is simplified. No transactions are ever provided by CometBFT, + and no transactions can ever be left in the mempool when CometBFT calls `PrepareProposal`: + the application trivially has all the information. +- UX is improved compared to how this can be done prior to this ADR. + +### Negative + +- With the `nop` mempool, it is up to the application to provide users with a way + to submit transactions and deliver those transactions to validators. + This is a considerable endeavor, and more basic appchains may consider it is not worth the hassle. +- There is a risk of wasting resources by those nodes that have a misconfigured + mempool (bandwidth, CPU, memory, etc). If there are TXs submitted (incorrectly) + via CometBFT's RPC, but those TXs are never submitted (correctly via an + app-specific interface) to the App. As those TXs risk being there until the node + is stopped. Moreover, those TXs will be replied & proposed every single block. + App developers will need to keep this in mind and panic on `CheckTx` or + `PrepareProposal` with non-empty list of transactions. +- Optimizing block proposals by only including transaction IDs (e.g. TX hashes) is more difficult. + The ABCI app could do it by submitting TX hashes (rather than TXs themselves) + in `PrepareProposal`, and then having a mechanism for pulling TXs from the + network upon `FinalizeBlock`. + +[sdk-app-mempool]: https://docs.cosmos.network/v0.47/build/building-apps/app-mempool +[adr-110]: https://github.com/cometbft/cometbft/pull/1565 +[HT94]: https://dl.acm.org/doi/book/10.5555/866693 +[cat-mempool]: https://github.com/cometbft/cometbft/pull/1472 \ No newline at end of file diff --git a/cometbft/v0.38/docs/architecture/adr-template.md b/cometbft/v0.38/docs/architecture/adr-template.md new file mode 100644 index 00000000..07ee2941 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/adr-template.md @@ -0,0 +1,101 @@ +# ADR {ADR-NUMBER}: {TITLE} + +## Changelog + +- {date}: {changelog} + +## Status + +> An architecture decision is considered "proposed" when a PR containing the ADR +> is submitted. When merged, an ADR must have a status associated with it, which +> must be one of: "Accepted", "Rejected", "Deprecated" or "Superseded". +> +> An accepted ADR's implementation status must be tracked via a tracking issue, +> milestone or project board (only one of these is necessary). For example: +> +> Accepted +> +> [Tracking issue](https://github.com/cometbft/cometbft/issues/123) +> [Milestone](https://github.com/cometbft/cometbft/milestones/123) +> [Project board](https://github.com/orgs/cometbft/projects/123) +> +> Rejected ADRs are captured as a record of recommendations that we specifically +> do not (and possibly never) want to implement. The ADR itself must, for +> posterity, include reasoning as to why it was rejected. +> +> If an ADR is deprecated, simply write "Deprecated" in this section. If an ADR +> is superseded by one or more other ADRs, provide local a reference to those +> ADRs, e.g.: +> +> Superseded by [ADR 123](./adr-123.md) + +Accepted | Rejected | Deprecated | Superseded by + +## Context + +> This section contains all the context one needs to understand the current state, +> and why there is a problem. It should be as succinct as possible and introduce +> the high level idea behind the solution. + +## Alternative Approaches + +> This section contains information around alternative options that are considered +> before making a decision. It should contain a explanation on why the alternative +> approach(es) were not chosen. + +## Decision + +> This section records the decision that was made. +> It is best to record as much info as possible from the discussion that happened. +> This aids in not having to go back to the Pull Request to get the needed information. + +## Detailed Design + +> This section does not need to be filled in at the start of the ADR, but must +> be completed prior to the merging of the implementation. +> +> Here are some common questions that get answered as part of the detailed design: +> +> - What are the user requirements? +> +> - What systems will be affected? +> +> - What new data structures are needed, what data structures will be changed? +> +> - What new APIs will be needed, what APIs will be changed? +> +> - What are the efficiency considerations (time/space)? +> +> - What are the expected access patterns (load/throughput)? +> +> - Are there any logging, monitoring or observability needs? +> +> - Are there any security considerations? +> +> - Are there any privacy considerations? +> +> - How will the changes be tested? +> +> - If the change is large, how will the changes be broken up for ease of review? +> +> - Will these changes require a breaking (major) release? +> +> - Does this change require coordination with the SDK or other? + +## Consequences + +> This section describes the consequences, after applying the decision. All +> consequences should be summarized here, not just the "positive" ones. + +### Positive + +### Negative + +### Neutral + +## References + +> Are there any relevant PR comments, issues that led up to this, or articles +> referenced for why we made the given design choice? If so link them here! + +- {reference link} diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/README.md b/cometbft/v0.38/docs/architecture/tendermint-core/README.md new file mode 100644 index 00000000..d1d8c276 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/README.md @@ -0,0 +1,105 @@ +--- +order: 1 +parent: + order: false +--- + +# Tendermint Core Architecture Decision Records (ADR) + +Here we record all high-level architecture decisions in the Tendermint Core +project. All implemented ADRs in this list naturally affect CometBFT, since +CometBFT is a fork of Tendermint Core as of December 2022. + +This list is currently frozen and kept for reference purposes. To add new ADRs, +please do so for CometBFT [here](../). + +## Table of Contents + +### Implemented + +- [ADR-001: Logging](./adr-001-logging.md) +- [ADR-002: Event-Subscription](./adr-002-event-subscription.md) +- [ADR-003: ABCI-APP-RPC](./adr-003-abci-app-rpc.md) +- [ADR-004: Historical-Validators](./adr-004-historical-validators.md) +- [ADR-005: Consensus-Params](./adr-005-consensus-params.md) +- [ADR-008: Priv-Validator](./adr-008-priv-validator.md) +- [ADR-009: ABCI-Design](./adr-009-ABCI-design.md) +- [ADR-010: Crypto-Changes](./adr-010-crypto-changes.md) +- [ADR-011: Monitoring](./adr-011-monitoring.md) +- [ADR-014: Secp-Malleability](./adr-014-secp-malleability.md) +- [ADR-015: Crypto-Encoding](./adr-015-crypto-encoding.md) +- [ADR-016: Protocol-Versions](./adr-016-protocol-versions.md) +- [ADR-017: Chain-Versions](./adr-017-chain-versions.md) +- [ADR-018: ABCI-Validators](./adr-018-ABCI-Validators.md) +- [ADR-019: Multisigs](./adr-019-multisigs.md) +- [ADR-020: Block-Size](./adr-020-block-size.md) +- [ADR-021: ABCI-Events](./adr-021-abci-events.md) +- [ADR-025: Commit](./adr-025-commit.md) +- [ADR-026: General-Merkle-Proof](./adr-026-general-merkle-proof.md) +- [ADR-033: Pubsub](./adr-033-pubsub.md) +- [ADR-034: Priv-Validator-File-Structure](./adr-034-priv-validator-file-structure.md) +- [ADR-043: Blockchain-RiRi-Org](./adr-043-blockchain-riri-org.md) +- [ADR-044: Lite-Client-With-Weak-Subjectivity](./adr-044-lite-client-with-weak-subjectivity.md) +- [ADR-046: Light-Client-Implementation](./adr-046-light-client-implementation.md) +- [ADR-047: Handling-Evidence-From-Light-Client](./adr-047-handling-evidence-from-light-client.md) +- [ADR-051: Double-Signing-Risk-Reduction](./adr-051-double-signing-risk-reduction.md) +- [ADR-052: Tendermint-Mode](./adr-052-tendermint-mode.md) +- [ADR-053: State-Sync-Prototype](./adr-053-state-sync-prototype.md) +- [ADR-054: Crypto-Encoding-2](./adr-054-crypto-encoding-2.md) +- [ADR-055: Protobuf-Design](./adr-055-protobuf-design.md) +- [ADR-056: Light-Client-Amnesia-Attacks](./adr-056-light-client-amnesia-attacks.md) +- [ADR-059: Evidence-Composition-and-Lifecycle](./adr-059-evidence-composition-and-lifecycle.md) +- [ADR-065: Custom Event Indexing](./adr-065-custom-event-indexing.md) +- [ADR-066: E2E-Testing](./adr-066-e2e-testing.md) +- [ADR-072: Restore Requests for Comments](./adr-072-request-for-comments.md) +- [ADR-076: Combine Spec and Tendermint Repositories](./adr-076-combine-spec-repo.md) +- [ADR-077: Configurable Block Retention](./adr-077-block-retention.md) +- [ADR-078: Non-zero Genesis](./adr-078-nonzero-genesis.md) + +### Accepted + +- [ADR-006: Trust-Metric](./adr-006-trust-metric.md) +- [ADR-024: Sign-Bytes](./adr-024-sign-bytes.md) +- [ADR-039: Peer-Behaviour](./adr-039-peer-behaviour.md) +- [ADR-063: Privval-gRPC](./adr-063-privval-grpc.md) +- [ADR-067: Mempool Refactor](./adr-067-mempool-refactor.md) +- [ADR-071: Proposer-Based Timestamps](./adr-071-proposer-based-timestamps.md) +- [ADR-075: RPC Event Subscription Interface](./adr-075-rpc-subscription.md) +- [ADR-079: Ed25519 Verification](./adr-079-ed25519-verification.md) +- [ADR-081: Protocol Buffers Management](./adr-081-protobuf-mgmt.md) + +### Deprecated + +- [ADR-035: Documentation](./adr-035-documentation.md) + +### Rejected + +- [ADR-023: ABCI-Propose-tx](./adr-023-ABCI-propose-tx.md) +- [ADR-029: Check-Tx-Consensus](./adr-029-check-tx-consensus.md) +- [ADR-058: Event-Hashing](./adr-058-event-hashing.md) + +### Proposed + +- [ADR-007: Trust-Metric-Usage](./adr-007-trust-metric-usage.md) +- [ADR-012: Peer-Transport](./adr-012-peer-transport.md) +- [ADR-013: Symmetric-Crypto](./adr-013-symmetric-crypto.md) +- [ADR-022: ABCI-Errors](./adr-022-abci-errors.md) +- [ADR-030: Consensus-Refactor](./adr-030-consensus-refactor.md) +- [ADR-036: Empty Blocks via ABCI](./adr-036-empty-blocks-abci.md) +- [ADR-037: Deliver-Block](./adr-037-deliver-block.md) +- [ADR-038: Non-Zero-Start-Height](./adr-038-non-zero-start-height.md) +- [ADR-040: Blockchain Reactor Refactor](./adr-040-blockchain-reactor-refactor.md) +- [ADR-041: Proposer-Selection-via-ABCI](./adr-041-proposer-selection-via-abci.md) +- [ADR-042: State Sync Design](./adr-042-state-sync.md) +- [ADR-045: ABCI-Evidence](./adr-045-abci-evidence.md) +- [ADR-050: Improved Trusted Peering](./adr-050-improved-trusted-peering.md) +- [ADR-057: RPC](./adr-057-RPC.md) +- [ADR-060: Go-API-Stability](./adr-060-go-api-stability.md) +- [ADR-061: P2P-Refactor-Scope](./adr-061-p2p-refactor-scope.md) +- [ADR-062: P2P-Architecture](./adr-062-p2p-architecture.md) +- [ADR-064: Batch Verification](./adr-064-batch-verification.md) +- [ADR-068: Reverse-Sync](./adr-068-reverse-sync.md) +- [ADR-069: Node Initialization](./adr-069-flexible-node-initialization.md) +- [ADR-073: Adopt LibP2P](./adr-073-libp2p.md) +- [ADR-074: Migrate Timeout Parameters to Consensus Parameters](./adr-074-timeout-params.md) +- [ADR-080: Reverse Sync](./adr-080-reverse-sync.md) diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-001-logging.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-001-logging.md new file mode 100644 index 00000000..b5df8bf7 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-001-logging.md @@ -0,0 +1,216 @@ +# ADR 1: Logging + +## Context + +Current logging system in Tendermint is very static and not flexible enough. + +Issues: [358](https://github.com/tendermint/tendermint/issues/358), [375](https://github.com/tendermint/tendermint/issues/375). + +What we want from the new system: + +- per package dynamic log levels +- dynamic logger setting (logger tied to the processing struct) +- conventions +- be more visually appealing + +"dynamic" here means the ability to set smth in runtime. + +## Decision + +### 1) An interface + +First, we will need an interface for all of our libraries (`tmlibs`, Tendermint, etc.). My personal preference is go-kit `Logger` interface (see Appendix A.), but that is too much a bigger change. Plus we will still need levels. + +```go +# log.go +type Logger interface { + Debug(msg string, keyvals ...interface{}) error + Info(msg string, keyvals ...interface{}) error + Error(msg string, keyvals ...interface{}) error + + With(keyvals ...interface{}) Logger +} +``` + +On a side note: difference between `Info` and `Notice` is subtle. We probably +could do without `Notice`. Don't think we need `Panic` or `Fatal` as a part of +the interface. These funcs could be implemented as helpers. In fact, we already +have some in `tmlibs/common`. + +- `Debug` - extended output for devs +- `Info` - all that is useful for a user +- `Error` - errors + +`Notice` should become `Info`, `Warn` either `Error` or `Debug` depending on the message, `Crit` -> `Error`. + +This interface should go into `tmlibs/log`. All libraries which are part of the core (tendermint/tendermint) should obey it. + +### 2) Logger with our current formatting + +On top of this interface, we will need to implement a stdout logger, which will be used when Tendermint is configured to output logs to STDOUT. + +Many people say that they like the current output, so let's stick with it. + +``` +NOTE[2017-04-25|14:45:08] ABCI Replay Blocks module=consensus appHeight=0 storeHeight=0 stateHeight=0 +``` + +Couple of minor changes: + +``` +I[2017-04-25|14:45:08.322] ABCI Replay Blocks module=consensus appHeight=0 storeHeight=0 stateHeight=0 +``` + +Notice the level is encoded using only one char plus milliseconds. + +Note: there are many other formats out there like [logfmt](https://brandur.org/logfmt). + +This logger could be implemented using any logger - [logrus](https://github.com/sirupsen/logrus), [go-kit/log](https://github.com/go-kit/kit/tree/master/log), [zap](https://github.com/uber-go/zap), log15 so far as it + +a) supports coloring output
+b) is moderately fast (buffering)
+c) conforms to the new interface or adapter could be written for it
+d) is somewhat configurable
+ +go-kit is my favorite so far. Check out how easy it is to color errors in red https://github.com/go-kit/kit/blob/master/log/term/example_test.go#L12. Although, coloring could only be applied to the whole string :( + +``` +go-kit +: flexible, modular +go-kit “-”: logfmt format https://brandur.org/logfmt + +logrus +: popular, feature rich (hooks), API and output is more like what we want +logrus -: not so flexible +``` + +```go +# tm_logger.go +// NewTmLogger returns a logger that encodes keyvals to the Writer in +// tm format. +func NewTmLogger(w io.Writer) Logger { + return &tmLogger{kitlog.NewLogfmtLogger(w)} +} + +func (l tmLogger) SetLevel(level string() { + switch (level) { + case "debug": + l.sourceLogger = level.NewFilter(l.sourceLogger, level.AllowDebug()) + } +} + +func (l tmLogger) Info(msg string, keyvals ...interface{}) error { + l.sourceLogger.Log("msg", msg, keyvals...) +} + +# log.go +func With(logger Logger, keyvals ...interface{}) Logger { + kitlog.With(logger.sourceLogger, keyvals...) +} +``` + +Usage: + +```go +logger := log.NewTmLogger(os.Stdout) +logger.SetLevel(config.GetString("log_level")) +node.SetLogger(log.With(logger, "node", Name)) +``` + +**Other log formatters** + +In the future, we may want other formatters like JSONFormatter. + +``` +{ "level": "notice", "time": "2017-04-25 14:45:08.562471297 -0400 EDT", "module": "consensus", "msg": "ABCI Replay Blocks", "appHeight": 0, "storeHeight": 0, "stateHeight": 0 } +``` + +### 3) Dynamic logger setting + +https://dave.cheney.net/2017/01/23/the-package-level-logger-anti-pattern + +This is the hardest part and where the most work will be done. logger should be tied to the processing struct, or the context if it adds some fields to the logger. + +```go +type BaseService struct { + log log15.Logger + name string + started uint32 // atomic + stopped uint32 // atomic +... +} +``` + +BaseService already contains `log` field, so most of the structs embedding it should be fine. We should rename it to `logger`. + +The only thing missing is the ability to set logger: + +``` +func (bs *BaseService) SetLogger(l log.Logger) { + bs.logger = l +} +``` + +### 4) Conventions + +Important keyvals should go first. Example: + +``` +correct +I[2017-04-25|14:45:08.322] ABCI Replay Blocks module=consensus instance=1 appHeight=0 storeHeight=0 stateHeight=0 +``` + +not + +``` +wrong +I[2017-04-25|14:45:08.322] ABCI Replay Blocks module=consensus appHeight=0 storeHeight=0 stateHeight=0 instance=1 +``` + +for that in most cases you'll need to add `instance` field to a logger upon creating, not when u log a particular message: + +```go +colorFn := func(keyvals ...interface{}) term.FgBgColor { + for i := 1; i < len(keyvals); i += 2 { + if keyvals[i] == "instance" && keyvals[i+1] == "1" { + return term.FgBgColor{Fg: term.Blue} + } else if keyvals[i] == "instance" && keyvals[i+1] == "1" { + return term.FgBgColor{Fg: term.Red} + } + } + return term.FgBgColor{} + } +logger := term.NewLogger(os.Stdout, log.NewTmLogger, colorFn) + +c1 := NewConsensusReactor(...) +c1.SetLogger(log.With(logger, "instance", 1)) + +c2 := NewConsensusReactor(...) +c2.SetLogger(log.With(logger, "instance", 2)) +``` + +## Status + +Implemented + +## Consequences + +### Positive + +Dynamic logger, which could be turned off for some modules at runtime. Public interface for other projects using Tendermint libraries. + +### Negative + +We may loose the ability to color keys in keyvalue pairs. go-kit allow you to easily change foreground / background colors of the whole string, but not its parts. + +### Neutral + +## Appendix A. + +I really like a minimalistic approach go-kit took with his logger https://github.com/go-kit/kit/tree/master/log: + +``` +type Logger interface { + Log(keyvals ...interface{}) error +} +``` + +See [The Hunt for a Logger Interface](https://web.archive.org/web/20210902161539/https://go-talks.appspot.com/github.com/ChrisHines/talks/structured-logging/structured-logging.slide#1). The advantage is greater composability (check out how go-kit defines colored logging or log-leveled logging on top of this interface https://github.com/go-kit/kit/tree/master/log). diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-002-event-subscription.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-002-event-subscription.md new file mode 100644 index 00000000..4cc2affb --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-002-event-subscription.md @@ -0,0 +1,88 @@ +# ADR 2: Event Subscription + +## Context + +In the light client (or any other client), the user may want to **subscribe to +a subset of transactions** (rather than all of them) using `/subscribe?event=X`. For +example, I want to subscribe for all transactions associated with a particular +account. Same for fetching. The user may want to **fetch transactions based on +some filter** (rather than fetching all the blocks). For example, I want to get +all transactions for a particular account in the last two weeks (`tx's block time >= '2017-06-05'`). + +Now you can't even subscribe to "all txs" in Tendermint. + +The goal is a simple and easy to use API for doing that. + +![Tx Send Flow Diagram](img/tags1.png) + +## Decision + +ABCI app return tags with a `DeliverTx` response inside the `data` field (_for +now, later we may create a separate field_). Tags is a list of key-value pairs, +protobuf encoded. + +Example data: + +```json +{ + "abci.account.name": "Igor", + "abci.account.address": "0xdeadbeef", + "tx.gas": 7 +} +``` + +### Subscribing for transactions events + +If the user wants to receive only a subset of transactions, ABCI-app must +return a list of tags with a `DeliverTx` response. These tags will be parsed and +matched with the current queries (subscribers). If the query matches the tags, +subscriber will get the transaction event. + +``` +/subscribe?query="tm.event = Tx AND tx.hash = AB0023433CF0334223212243BDD AND abci.account.invoice.number = 22" +``` + +A new package must be developed to replace the current `events` package. It +will allow clients to subscribe to a different types of events in the future: + +``` +/subscribe?query="abci.account.invoice.number = 22" +/subscribe?query="abci.account.invoice.owner CONTAINS Igor" +``` + +### Fetching transactions + +This is a bit tricky because a) we want to support a number of indexers, all of +which have a different API b) we don't know whenever tags will be sufficient +for the most apps (I guess we'll see). + +``` +/txs/search?query="tx.hash = AB0023433CF0334223212243BDD AND abci.account.owner CONTAINS Igor" +/txs/search?query="abci.account.owner = Igor" +``` + +For historic queries we will need a indexing storage (Postgres, SQLite, ...). + +### Issues + +- https://github.com/tendermint/tendermint/issues/376 +- https://github.com/tendermint/tendermint/issues/287 +- https://github.com/tendermint/tendermint/issues/525 (related) + +## Status + +Implemented + +## Consequences + +### Positive + +- same format for event notifications and search APIs +- powerful enough query + +### Negative + +- performance of the `match` function (where we have too many queries / subscribers) +- there is an issue where there are too many txs in the DB + +### Neutral diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-003-abci-app-rpc.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-003-abci-app-rpc.md new file mode 100644 index 00000000..3bc46498 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-003-abci-app-rpc.md @@ -0,0 +1,34 @@ +# ADR 3: Must an ABCI-app have an RPC server? + +## Context + +ABCI-server could expose its own RPC-server and act as a proxy to Tendermint. + +The idea was for the Tendermint RPC to just be a transparent proxy to the app. +Clients need to talk to Tendermint for proofs, unless we burden all app devs +with exposing Tendermint proof stuff. Also seems less complex to lock down one +server than two, but granted it makes querying a bit more kludgy since it needs +to be passed as a `Query`. Also, **having a very standard rpc interface means +the light-client can work with all apps and handle proofs**. The only +app-specific logic is decoding the binary data to a more readable form (eg. +json). This is a huge advantage for code-reuse and standardization. + +## Decision + +We dont expose an RPC server on any of our ABCI-apps. + +## Status + +Implemented + +## Consequences + +### Positive + +- Unified interface for all apps + +### Negative + +- `Query` interface + +### Neutral diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-004-historical-validators.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-004-historical-validators.md new file mode 100644 index 00000000..7341c473 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-004-historical-validators.md @@ -0,0 +1,38 @@ +# ADR 004: Historical Validators + +## Context + +Right now, we can query the present validator set, but there is no history. +If you were offline for a long time, there is no way to reconstruct past validators. This is needed for the light client and we agreed needs enhancement of the API. + +## Decision + +For every block, store a new structure that contains either the latest validator set, +or the height of the last block for which the validator set changed. Note this is not +the height of the block which returned the validator set change itself, but the next block, +ie. the first block it comes into effect for. + +Storing the validators will be handled by the `state` package. + +At some point in the future, we may consider more efficient storage in the case where the validators +are updated frequently - for instance by only saving the diffs, rather than the whole set. + +An alternative approach suggested keeping the validator set, or diffs of it, in a merkle IAVL tree. +While it might afford cheaper proofs that a validator set has not changed, it would be more complex, +and likely less efficient. + +## Status + +Implemented + +## Consequences + +### Positive + +- Can query old validator sets, with proof. + +### Negative + +- Writes an extra structure to disk with every block. + +### Neutral diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-005-consensus-params.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-005-consensus-params.md new file mode 100644 index 00000000..550e9fc0 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-005-consensus-params.md @@ -0,0 +1,85 @@ +# ADR 005: Consensus Params + +## Context + +Consensus critical parameters controlling blockchain capacity have until now been hard coded, loaded from a local config, or neglected. +Since they may be need to be different in different networks, and potentially to evolve over time within +networks, we seek to initialize them in a genesis file, and expose them through the ABCI. + +While we have some specific parameters now, like maximum block and transaction size, we expect to have more in the future, +such as a period over which evidence is valid, or the frequency of checkpoints. + +## Decision + +### ConsensusParams + +No consensus critical parameters should ever be found in the `config.toml`. + +A new `ConsensusParams` is optionally included in the `genesis.json` file, +and loaded into the `State`. Any items not included are set to their default value. +A value of 0 is undefined (see ABCI, below). A value of -1 is used to indicate the parameter does not apply. +The parameters are used to determine the validity of a block (and tx) via the union of all relevant parameters. + +``` +type ConsensusParams struct { + BlockSize + TxSize + BlockGossip +} + +type BlockSize struct { + MaxBytes int + MaxTxs int + MaxGas int +} + +type TxSize struct { + MaxBytes int + MaxGas int +} + +type BlockGossip struct { + BlockPartSizeBytes int +} +``` + +The `ConsensusParams` can evolve over time by adding new structs that cover different aspects of the consensus rules. + +The `BlockPartSizeBytes` and the `BlockSize.MaxBytes` are enforced to be greater than 0. +The former because we need a part size, the latter so that we always have at least some sanity check over the size of blocks. + +### ABCI + +#### InitChain + +InitChain currently takes the initial validator set. It should be extended to also take parts of the ConsensusParams. +There is some case to be made for it to take the entire Genesis, except there may be things in the genesis, +like the BlockPartSize, that the app shouldn't really know about. + +#### EndBlock + +The EndBlock response includes a `ConsensusParams`, which includes BlockSize and TxSize, but not BlockGossip. +Other param struct can be added to `ConsensusParams` in the future. +The `0` value is used to denote no change. +Any other value will update that parameter in the `State.ConsensusParams`, to be applied for the next block. +Tendermint should have hard-coded upper limits as sanity checks. + +## Status + +Implemented + +## Consequences + +### Positive + +- Alternative capacity limits and consensus parameters can be specified without re-compiling the software. +- They can also change over time under the control of the application + +### Negative + +- More exposed parameters is more complexity +- Different rules at different heights in the blockchain complicates fast sync + +### Neutral + +- The TxSize, which checks validity, may be in conflict with the config's `max_block_size_tx`, which determines proposal sizes diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-006-trust-metric.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-006-trust-metric.md new file mode 100644 index 00000000..60897820 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-006-trust-metric.md @@ -0,0 +1,229 @@ +# ADR 006: Trust Metric Design + +## Context + +The proposed trust metric will allow Tendermint to maintain local trust rankings for peers it has directly interacted with, which can then be used to implement soft security controls. The calculations were obtained from the [TrustGuard](https://dl.acm.org/citation.cfm?id=1060808) project. + +### Background + +The Tendermint Core project developers would like to improve Tendermint security and reliability by keeping track of the level of trustworthiness peers have demonstrated within the peer-to-peer network. This way, undesirable outcomes from peers will not immediately result in them being dropped from the network (potentially causing drastic changes to take place). Instead, peers behavior can be monitored with appropriate metrics and be removed from the network once Tendermint Core is certain the peer is a threat. For example, when the PEXReactor makes a request for peers network addresses from a already known peer, and the returned network addresses are unreachable, this untrustworthy behavior should be tracked. Returning a few bad network addresses probably shouldn’t cause a peer to be dropped, while excessive amounts of this behavior does qualify the peer being dropped. + +Trust metrics can be circumvented by malicious nodes through the use of strategic oscillation techniques, which adapts the malicious node’s behavior pattern in order to maximize its goals. For instance, if the malicious node learns that the time interval of the Tendermint trust metric is _X_ hours, then it could wait _X_ hours in-between malicious activities. We could try to combat this issue by increasing the interval length, yet this will make the system less adaptive to recent events. + +Instead, having shorter intervals, but keeping a history of interval values, will give our metric the flexibility needed in order to keep the network stable, while also making it resilient against a strategic malicious node in the Tendermint peer-to-peer network. Also, the metric can access trust data over a rather long period of time while not greatly increasing its history size by aggregating older history values over a larger number of intervals, and at the same time, maintain great precision for the recent intervals. This approach is referred to as fading memories, and closely resembles the way human beings remember their experiences. The trade-off to using history data is that the interval values should be preserved in-between executions of the node. + +### References + +S. Mudhakar, L. Xiong, and L. Liu, “TrustGuard: Countering Vulnerabilities in Reputation Management for Decentralized Overlay Networks,” in _Proceedings of the 14th international conference on World Wide Web, pp. 422-431_, May 2005. + +## Decision + +The proposed trust metric will allow a developer to inform the trust metric store of all good and bad events relevant to a peer's behavior, and at any time, the metric can be queried for a peer's current trust ranking. + +The three subsections below will cover the process being considered for calculating the trust ranking, the concept of the trust metric store, and the interface for the trust metric. + +### Proposed Process + +The proposed trust metric will count good and bad events relevant to the object, and calculate the percent of counters that are good over an interval with a predefined duration. This is the procedure that will continue for the life of the trust metric. When the trust metric is queried for the current **trust value**, a resilient equation will be utilized to perform the calculation. + +The equation being proposed resembles a Proportional-Integral-Derivative (PID) controller used in control systems. The proportional component allows us to be sensitive to the value of the most recent interval, while the integral component allows us to incorporate trust values stored in the history data, and the derivative component allows us to give weight to sudden changes in the behavior of a peer. We compute the trust value of a peer in interval i based on its current trust ranking, its trust rating history prior to interval _i_ (over the past _maxH_ number of intervals) and its trust ranking fluctuation. We will break up the equation into the three components. + +```math +(1) Proportional Value = a * R[i] +``` + +where _R_[*i*] denotes the raw trust value at time interval _i_ (where _i_ == 0 being current time) and _a_ is the weight applied to the contribution of the current reports. The next component of our equation uses a weighted sum over the last _maxH_ intervals to calculate the history value for time _i_: + +`H[i] =` ![formula1](img/formula1.png "Weighted Sum Formula") + +The weights can be chosen either optimistically or pessimistically. An optimistic weight creates larger weights for newer history data values, while the the pessimistic weight creates larger weights for time intervals with lower scores. The default weights used during the calculation of the history value are optimistic and calculated as _Wk_ = 0.8^_k_, for time interval _k_. With the history value available, we can now finish calculating the integral value: + +```math +(2) Integral Value = b * H[i] +``` + +Where _H_[*i*] denotes the history value at time interval _i_ and _b_ is the weight applied to the contribution of past performance for the object being measured. The derivative component will be calculated as follows: + +```math +D[i] = R[i] – H[i] + +(3) Derivative Value = c(D[i]) * D[i] +``` + +Where the value of _c_ is selected based on the _D_[*i*] value relative to zero. The default selection process makes _c_ equal to 0 unless _D_[*i*] is a negative value, in which case c is equal to 1. The result is that the maximum penalty is applied when current behavior is lower than previously experienced behavior. If the current behavior is better than the previously experienced behavior, then the Derivative Value has no impact on the trust value. With the three components brought together, our trust value equation is calculated as follows: + +```math +TrustValue[i] = a * R[i] + b * H[i] + c(D[i]) * D[i] +``` + +As a performance optimization that will keep the amount of raw interval data being saved to a reasonable size of _m_, while allowing us to represent 2^_m_ - 1 history intervals, we can employ the fading memories technique that will trade space and time complexity for the precision of the history data values by summarizing larger quantities of less recent values. While our equation above attempts to access up to _maxH_ (which can be 2^_m_ - 1), we will map those requests down to _m_ values using equation 4 below: + +```math +(4) j = index, where index > 0 +``` + +Where _j_ is one of _(0, 1, 2, … , m – 1)_ indices used to access history interval data. Now we can access the raw intervals using the following calculations: + +```math +R[0] = raw data for current time interval +``` + +`R[j] =` ![formula2](img/formula2.png "Fading Memories Formula") + +### Trust Metric Store + +Similar to the P2P subsystem AddrBook, the trust metric store will maintain information relevant to Tendermint peers. Additionally, the trust metric store will ensure that trust metrics will only be active for peers that a node is currently and directly engaged with. + +Reactors will provide a peer key to the trust metric store in order to retrieve the associated trust metric. The trust metric can then record new positive and negative events experienced by the reactor, as well as provided the current trust score calculated by the metric. + +When the node is shutting down, the trust metric store will save history data for trust metrics associated with all known peers. This saved information allows experiences with a peer to be preserved across node executions, which can span a tracking windows of days or weeks. The trust history data is loaded automatically during OnStart. + +### Interface Detailed Design + +Each trust metric allows for the recording of positive/negative events, querying the current trust value/score, and the stopping/pausing of tracking over time intervals. This can be seen below: + +```go +// TrustMetric - keeps track of peer reliability +type TrustMetric struct { + // Private elements. +} + +// Pause tells the metric to pause recording data over time intervals. +// All method calls that indicate events will unpause the metric +func (tm *TrustMetric) Pause() {} + +// Stop tells the metric to stop recording data over time intervals +func (tm *TrustMetric) Stop() {} + +// BadEvents indicates that an undesirable event(s) took place +func (tm *TrustMetric) BadEvents(num int) {} + +// GoodEvents indicates that a desirable event(s) took place +func (tm *TrustMetric) GoodEvents(num int) {} + +// TrustValue gets the dependable trust value; always between 0 and 1 +func (tm *TrustMetric) TrustValue() float64 {} + +// TrustScore gets a score based on the trust value always between 0 and 100 +func (tm *TrustMetric) TrustScore() int {} + +// NewMetric returns a trust metric with the default configuration +func NewMetric() *TrustMetric {} + +//------------------------------------------------------------------------------------------------ +// For example + +tm := NewMetric() + +tm.BadEvents(1) +score := tm.TrustScore() + +tm.Stop() +``` + +Some of the trust metric parameters can be configured. The weight values should probably be left alone in more cases, yet the time durations for the tracking window and individual time interval should be considered. + +```go +// TrustMetricConfig - Configures the weight functions and time intervals for the metric +type TrustMetricConfig struct { + // Determines the percentage given to current behavior + ProportionalWeight float64 + + // Determines the percentage given to prior behavior + IntegralWeight float64 + + // The window of time that the trust metric will track events across. + // This can be set to cover many days without issue + TrackingWindow time.Duration + + // Each interval should be short for adapability. + // Less than 30 seconds is too sensitive, + // and greater than 5 minutes will make the metric numb + IntervalLength time.Duration +} + +// DefaultConfig returns a config with values that have been tested and produce desirable results +func DefaultConfig() TrustMetricConfig {} + +// NewMetricWithConfig returns a trust metric with a custom configuration +func NewMetricWithConfig(tmc TrustMetricConfig) *TrustMetric {} + +//------------------------------------------------------------------------------------------------ +// For example + +config := TrustMetricConfig{ + TrackingWindow: time.Minute * 60 * 24, // one day + IntervalLength: time.Minute * 2, +} + +tm := NewMetricWithConfig(config) + +tm.BadEvents(10) +tm.Pause() +tm.GoodEvents(1) // becomes active again +``` + +A trust metric store should be created with a DB that has persistent storage so it can save history data across node executions. All trust metrics instantiated by the store will be created with the provided TrustMetricConfig configuration. + +When you attempt to fetch the trust metric for a peer, and an entry does not exist in the trust metric store, a new metric is automatically created and the entry made within the store. + +In additional to the fetching method, GetPeerTrustMetric, the trust metric store provides a method to call when a peer has disconnected from the node. This is so the metric can be paused (history data will not be saved) for periods of time when the node is not having direct experiences with the peer. + +```go +// TrustMetricStore - Manages all trust metrics for peers +type TrustMetricStore struct { + cmn.BaseService + + // Private elements +} + +// OnStart implements Service +func (tms *TrustMetricStore) OnStart(context.Context) error { return nil } + +// OnStop implements Service +func (tms *TrustMetricStore) OnStop() {} + +// NewTrustMetricStore returns a store that saves data to the DB +// and uses the config when creating new trust metrics +func NewTrustMetricStore(db dbm.DB, tmc TrustMetricConfig) *TrustMetricStore {} + +// Size returns the number of entries in the trust metric store +func (tms *TrustMetricStore) Size() int {} + +// GetPeerTrustMetric returns a trust metric by peer key +func (tms *TrustMetricStore) GetPeerTrustMetric(key string) *TrustMetric {} + +// PeerDisconnected pauses the trust metric associated with the peer identified by the key +func (tms *TrustMetricStore) PeerDisconnected(key string) {} + +//------------------------------------------------------------------------------------------------ +// For example + +db := dbm.NewDB("trusthistory", "goleveldb", dirPathStr) +tms := NewTrustMetricStore(db, DefaultConfig()) + +tm := tms.GetPeerTrustMetric(key) +tm.BadEvents(1) + +tms.PeerDisconnected(key) +``` + +## Status + +Approved. + +## Consequences + +### Positive + +- The trust metric will allow Tendermint to make non-binary security and reliability decisions +- Will help Tendermint implement deterrents that provide soft security controls, yet avoids disruption on the network +- Will provide useful profiling information when analyzing performance over time related to peer interaction + +### Negative + +- Requires saving the trust metric history data across node executions + +### Neutral + +- Keep in mind that, good events need to be recorded just as bad events do using this implementation diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-007-trust-metric-usage.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-007-trust-metric-usage.md new file mode 100644 index 00000000..0daa4bb3 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-007-trust-metric-usage.md @@ -0,0 +1,106 @@ +# ADR 007: Trust Metric Usage Guide + +## Context + +Tendermint is required to monitor peer quality in order to inform its peer dialing and peer exchange strategies. + +When a node first connects to the network, it is important that it can quickly find good peers. +Thus, while a node has fewer connections, it should prioritize connecting to higher quality peers. +As the node becomes well connected to the rest of the network, it can dial lesser known or lesser +quality peers and help assess their quality. Similarly, when queried for peers, a node should make +sure they dont return low quality peers. + +Peer quality can be tracked using a trust metric that flags certain behaviors as good or bad. When enough +bad behavior accumulates, we can mark the peer as bad and disconnect. +For example, when the PEXReactor makes a request for peers network addresses from an already known peer, and the returned network addresses are unreachable, this undesirable behavior should be tracked. Returning a few bad network addresses probably shouldn’t cause a peer to be dropped, while excessive amounts of this behavior does qualify the peer for removal. The originally proposed approach and design document for the trust metric can be found in the [ADR 006](adr-006-trust-metric.md) document. + +The trust metric implementation allows a developer to obtain a peer's trust metric from a trust metric store, and track good and bad events relevant to a peer's behavior, and at any time, the peer's metric can be queried for a current trust value. The current trust value is calculated with a formula that utilizes current behavior, previous behavior, and change between the two. Current behavior is calculated as the percentage of good behavior within a time interval. The time interval is short; probably set between 30 seconds and 5 minutes. On the other hand, the historic data can estimate a peer's behavior over days worth of tracking. At the end of a time interval, the current behavior becomes part of the historic data, and a new time interval begins with the good and bad counters reset to zero. + +These are some important things to keep in mind regarding how the trust metrics handle time intervals and scoring: + +- Each new time interval begins with a perfect score +- Bad events quickly bring the score down and good events cause the score to slowly rise +- When the time interval is over, the percentage of good events becomes historic data. + +Some useful information about the inner workings of the trust metric: + +- When a trust metric is first instantiated, a timer (ticker) periodically fires in order to handle transitions between trust metric time intervals +- If a peer is disconnected from a node, the timer should be paused, since the node is no longer connected to that peer +- The ability to pause the metric is supported with the store **PeerDisconnected** method and the metric **Pause** method +- After a pause, if a good or bad event method is called on a metric, it automatically becomes unpaused and begins a new time interval. + +## Decision + +The trust metric capability is now available, yet, it still leaves the question of how should it be applied throughout Tendermint in order to properly track the quality of peers? + +### Proposed Process + +Peers are managed using an address book and a trust metric: + +- The address book keeps a record of peers and provides selection methods +- The trust metric tracks the quality of the peers + +#### Presence in Address Book + +Outbound peers are added to the address book before they are dialed, +and inbound peers are added once the peer connection is set up. +Peers are also added to the address book when they are received in response to +a pexRequestMessage. + +While a node has less than `needAddressThreshold`, it will periodically request more, +via pexRequestMessage, from randomly selected peers and from newly dialed outbound peers. + +When a new address is added to an address book that has more than `0.5*needAddressThreshold` addresses, +then with some low probability, a randomly chosen low quality peer is removed. + +#### Outbound Peers + +Peers attempt to maintain a minimum number of outbound connections by +repeatedly querying the address book for peers to connect to. +While a node has few to no outbound connections, the address book is biased to return +higher quality peers. As the node increases the number of outbound connections, +the address book is biased to return less-vetted or lower-quality peers. + +#### Inbound Peers + +Peers also maintain a maximum number of total connections, MaxNumPeers. +If a peer has MaxNumPeers, new incoming connections will be accepted with low probability. +When such a new connection is accepted, the peer disconnects from a probabilistically chosen low ranking peer +so it does not exceed MaxNumPeers. + +#### Peer Exchange + +When a peer receives a pexRequestMessage, it returns a random sample of high quality peers from the address book. Peers with no score or low score should not be inclided in a response to pexRequestMessage. + +#### Peer Quality + +Peer quality is tracked in the connection and across the reactors by storing the TrustMetric in the peer's +thread safe Data store. + +Peer behavior is then defined as one of the following: + +- Fatal - something outright malicious that causes us to disconnect the peer and ban it from the address book for some amount of time +- Bad - Any kind of timeout, messages that don't unmarshal, fail other validity checks, or messages we didn't ask for or aren't expecting (usually worth one bad event) +- Neutral - Unknown channels/message types/version upgrades (no good or bad events recorded) +- Correct - Normal correct behavior (worth one good event) +- Good - some random majority of peers per reactor sending us useful messages (worth more than one good event). + +Note that Fatal behavior causes us to remove the peer, and neutral behavior does not affect the score. + +## Status + +Proposed. + +## Consequences + +### Positive + +- Bringing the address book and trust metric store together will cause the network to be built in a way that encourages greater security and reliability. + +### Negative + +- TBD + +### Neutral + +- Keep in mind that, good events need to be recorded just as bad events do using this implementation. diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-008-priv-validator.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-008-priv-validator.md new file mode 100644 index 00000000..a3d31048 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-008-priv-validator.md @@ -0,0 +1,35 @@ +# ADR 008: SocketPV + +Tendermint node's should support only two in-process PrivValidator +implementations: + +- FilePV uses an unencrypted private key in a "priv_validator.json" file - no + configuration required (just `tendermint init validator`). +- TCPVal and IPCVal use TCP and Unix sockets respectively to send signing requests + to another process - the user is responsible for starting that process themselves. + +Both TCPVal and IPCVal addresses can be provided via flags at the command line +or in the configuration file; TCPVal addresses must be of the form +`tcp://:` and IPCVal addresses `unix:///path/to/file.sock` - +doing so will cause Tendermint to ignore any private validator files. + +TCPVal will listen on the given address for incoming connections from an external +private validator process. It will halt any operation until at least one external +process successfully connected. + +The external priv_validator process will dial the address to connect to +Tendermint, and then Tendermint will send requests on the ensuing connection to +sign votes and proposals. Thus the external process initiates the connection, +but the Tendermint process makes all requests. In a later stage we're going to +support multiple validators for fault tolerance. To prevent double signing they +need to be synced, which is deferred to an external solution (see #1185). + +Conversely, IPCVal will make an outbound connection to an existing socket opened +by the external validator process. + +In addition, Tendermint will provide implementations that can be run in that +external process. These include: + +- FilePV will encrypt the private key, and the user must enter password to + decrypt key when process is started. +- LedgerPV uses a Ledger Nano S to handle all signing. diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-009-ABCI-design.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-009-ABCI-design.md new file mode 100644 index 00000000..3876ffb3 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-009-ABCI-design.md @@ -0,0 +1,271 @@ +# ADR 009: ABCI UX Improvements + +## Changelog + +23-06-2018: Some minor fixes from review +07-06-2018: Some updates based on discussion with Jae +07-06-2018: Initial draft to match what was released in ABCI v0.11 + +## Context + +The ABCI was first introduced in late 2015. It's purpose is to be: + +- a generic interface between state machines and their replication engines +- agnostic to the language the state machine is written in +- agnostic to the replication engine that drives it + +This means ABCI should provide an interface for both pluggable applications and +pluggable consensus engines. + +To achieve this, it uses Protocol Buffers (proto3) for message types. The dominant +implementation is in Go. + +After some recent discussions with the community on github, the following were +identified as pain points: + +- Amino encoded types +- Managing validator sets +- Imports in the protobuf file + +See the [references](#references) for more. + +### Imports + +The native proto library in Go generates inflexible and verbose code. +Many in the Go community have adopted a fork called +[gogoproto](https://github.com/cosmos/gogoproto) that provides a +variety of features aimed to improve the developer experience. +While `gogoproto` is nice, it creates an additional dependency, and compiling +the protobuf types for other languages has been reported to fail when `gogoproto` is used. + +### Amino + +Amino is an encoding protocol designed to improve over insufficiencies of protobuf. +It's goal is to be proto4. + +Many people are frustrated by incompatibility with protobuf, +and with the requirement for Amino to be used at all within ABCI. + +We intend to make Amino successful enough that we can eventually use it for ABCI +message types directly. By then it should be called proto4. In the meantime, +we want it to be easy to use. + +### PubKey + +PubKeys are encoded using Amino (and before that, go-wire). +Ideally, PubKeys are an interface type where we don't know all the +implementation types, so its unfitting to use `oneof` or `enum`. + +### Addresses + +The address for ED25519 pubkey is the RIPEMD160 of the Amino +encoded pubkey. This introduces an Amino dependency in the address generation, +a functionality that is widely required and should be easy to compute as +possible. + +### Validators + +To change the validator set, applications can return a list of validator updates +with ResponseEndBlock. In these updates, the public key _must_ be included, +because Tendermint requires the public key to verify validator signatures. This +means ABCI developers have to work with PubKeys. That said, it would also be +convenient to work with address information, and for it to be simple to do so. + +### AbsentValidators + +Tendermint also provides a list of validators in BeginBlock who did not sign the +last block. This allows applications to reflect availability behavior in the +application, for instance by punishing validators for not having votes included +in commits. + +### InitChain + +Tendermint passes in a list of validators here, and nothing else. It would +benefit the application to be able to control the initial validator set. For +instance the genesis file could include application-based information about the +initial validator set that the application could process to determine the +initial validator set. Additionally, InitChain would benefit from getting all +the genesis information. + +### Header + +ABCI provides the Header in RequestBeginBlock so the application can have +important information about the latest state of the blockchain. + +## Decision + +### Imports + +Move away from gogoproto. In the short term, we will just maintain a second +protobuf file without the gogoproto annotations. In the medium term, we will +make copies of all the structs in Golang and shuttle back and forth. In the long +term, we will use Amino. + +### Amino + +To simplify ABCI application development in the short term, +Amino will be completely removed from the ABCI: + +- It will not be required for PubKey encoding +- It will not be required for computing PubKey addresses + +That said, we are working to make Amino a huge success, and to become proto4. +To facilitate adoption and cross-language compatibility in the near-term, Amino +v1 will: + +- be fully compatible with the subset of proto3 that excludes `oneof` +- use the Amino prefix system to provide interface types, as opposed to `oneof` + style union types. + +That said, an Amino v2 will be worked on to improve the performance of the +format and its useability in cryptographic applications. + +### PubKey + +Encoding schemes infect software. As a generic middleware, ABCI aims to have +some cross scheme compatibility. For this it has no choice but to include opaque +bytes from time to time. While we will not enforce Amino encoding for these +bytes yet, we need to provide a type system. The simplest way to do this is to +use a type string. + +PubKey will now look like: + +``` +message PubKey { + string type + bytes data +} +``` + +where `type` can be: + +- "ed225519", with `data = ` +- "secp256k1", with `data = <33-byte OpenSSL compressed pubkey>` + +As we want to retain flexibility here, and since ideally, PubKey would be an +interface type, we do not use `enum` or `oneof`. + +### Addresses + +To simplify and improve computing addresses, we change it to the first 20-bytes of the SHA256 +of the raw 32-byte public key. + +We continue to use the Bitcoin address scheme for secp256k1 keys. + +### Validators + +Add a `bytes address` field: + +``` +message Validator { + bytes address + PubKey pub_key + int64 power +} +``` + +### RequestBeginBlock and AbsentValidators + +To simplify this, RequestBeginBlock will include the complete validator set, +including the address, and voting power of each validator, along +with a boolean for whether or not they voted: + +``` +message RequestBeginBlock { + bytes hash + Header header + LastCommitInfo last_commit_info + repeated Evidence byzantine_validators +} + +message LastCommitInfo { + int32 CommitRound + repeated SigningValidator validators +} + +message SigningValidator { + Validator validator + bool signed_last_block +} +``` + +Note that in Validators in RequestBeginBlock, we DO NOT include public keys. Public keys are +larger than addresses and in the future, with quantum computers, will be much +larger. The overhead of passing them, especially during fast-sync, is +significant. + +Additional, addresses are changing to be simpler to compute, further removing +the need to include pubkeys here. + +In short, ABCI developers must be aware of both addresses and public keys. + +### ResponseEndBlock + +Since ResponseEndBlock includes Validator, it must now include their address. + +### InitChain + +Change RequestInitChain to give the app all the information from the genesis file: + +``` +message RequestInitChain { + int64 time + string chain_id + ConsensusParams consensus_params + repeated Validator validators + bytes app_state_bytes +} +``` + +Change ResponseInitChain to allow the app to specify the initial validator set +and consensus parameters. + +``` +message ResponseInitChain { + ConsensusParams consensus_params + repeated Validator validators +} +``` + +### Header + +Now that Tendermint Amino will be compatible with proto3, the Header in ABCI +should exactly match the Tendermint header - they will then be encoded +identically in ABCI and in Tendermint Core. + +## Status + +Implemented + +## Consequences + +### Positive + +- Easier for developers to build on the ABCI +- ABCI and Tendermint headers are identically serialized + +### Negative + +- Maintenance overhead of alternative type encoding scheme +- Performance overhead of passing all validator info every block (at least its + only addresses, and not also pubkeys) +- Maintenance overhead of duplicate types + +### Neutral + +- ABCI developers must know about validator addresses + +## References + +- [ABCI v0.10.3 Specification (before this + proposal)](https://github.com/tendermint/abci/blob/v0.10.3/specification.rst) +- [ABCI v0.11.0 Specification (implementing first draft of this + proposal)](https://github.com/tendermint/abci/blob/v0.11.0/specification.md) +- [Ed25519 addresses](https://github.com/tendermint/go-crypto/issues/103) +- [InitChain contains the + Genesis](https://github.com/tendermint/abci/issues/216) +- [PubKeys](https://github.com/tendermint/tendermint/issues/1524) +- [Notes on + Header](https://github.com/tendermint/tendermint/issues/1605) +- [Gogoproto issues](https://github.com/tendermint/abci/issues/256) +- [Absent Validators](https://github.com/tendermint/abci/issues/231) diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-010-crypto-changes.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-010-crypto-changes.md new file mode 100644 index 00000000..41d15da3 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-010-crypto-changes.md @@ -0,0 +1,77 @@ +# ADR 010: Crypto Changes + +## Context + +Tendermint is a cryptographic protocol that uses and composes a variety of cryptographic primitives. + +After nearly 4 years of development, Tendermint has recently undergone multiple security reviews to search for vulnerabilities and to assess the the use and composition of cryptographic primitives. + +### Hash Functions + +Tendermint uses RIPEMD160 universally as a hash function, most notably in its Merkle tree implementation. + +RIPEMD160 was chosen because it provides the shortest fingerprint that is long enough to be considered secure (ie. birthday bound of 80-bits). +It was also developed in the open academic community, unlike NSA-designed algorithms like SHA256. + +That said, the cryptographic community appears to unanimously agree on the security of SHA256. It has become a universal standard, especially now that SHA1 is broken, being required in TLS connections and having optimized support in hardware. + +### Merkle Trees + +Tendermint uses a simple Merkle tree to compute digests of large structures like transaction batches +and even blockchain headers. The Merkle tree length prefixes byte arrays before concatenating and hashing them. +It uses RIPEMD160. + +### Addresses + +ED25519 addresses are computed using the RIPEMD160 of the Amino encoding of the public key. +RIPEMD160 is generally considered an outdated hash function, and is much slower +than more modern functions like SHA256 or Blake2. + +### Authenticated Encryption + +Tendermint P2P connections use authenticated encryption to provide privacy and authentication in the communications. +This is done using the simple Station-to-Station protocol with the NaCL Ed25519 library. + +While there have been no vulnerabilities found in the implementation, there are some concerns: + +- NaCL uses Salsa20, a not-widely used and relatively out-dated stream cipher that has been obsoleted by ChaCha20 +- Connections use RIPEMD160 to compute a value that is used for the encryption nonce with subtle requirements on how it's used + +## Decision + +### Hash Functions + +Use the first 20-bytes of the SHA256 hash instead of RIPEMD160 for everything + +### Merkle Trees + +TODO + +### Addresses + +Compute ED25519 addresses as the first 20-bytes of the SHA256 of the raw 32-byte public key + +### Authenticated Encryption + +Make the following changes: + +- Use xChaCha20 instead of xSalsa20 - https://github.com/tendermint/tendermint/issues/1124 +- Use an HKDF instead of RIPEMD160 to compute nonces - https://github.com/tendermint/tendermint/issues/1165 + +## Status + +Implemented + +## Consequences + +### Positive + +- More modern and standard cryptographic functions with wider adoption and hardware acceleration + +### Negative + +- Exact authenticated encryption construction isn't already provided in a well-used library + +### Neutral + +## References diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-011-monitoring.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-011-monitoring.md new file mode 100644 index 00000000..e4b62c26 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-011-monitoring.md @@ -0,0 +1,116 @@ +# ADR 011: Monitoring + +## Changelog + +08-06-2018: Initial draft +11-06-2018: Reorg after @xla comments +13-06-2018: Clarification about usage of labels + +## Context + +In order to bring more visibility into Tendermint, we would like it to report +metrics and, maybe later, traces of transactions and RPC queries. See +https://github.com/tendermint/tendermint/issues/986. + +A few solutions were considered: + +1. [Prometheus](https://prometheus.io) + a) Prometheus API + b) [go-kit metrics package](https://github.com/go-kit/kit/tree/master/metrics) as an interface plus Prometheus + c) [telegraf](https://github.com/influxdata/telegraf) + d) new service, which will listen to events emitted by pubsub and report metrics +2. [OpenCensus](https://opencensus.io/introduction/) + +### 1. Prometheus + +Prometheus seems to be the most popular product out there for monitoring. It has +a Go client library, powerful queries, alerts. + +**a) Prometheus API** + +We can commit to using Prometheus in Tendermint, but I think Tendermint users +should be free to choose whatever monitoring tool they feel will better suit +their needs (if they don't have existing one already). So we should try to +abstract interface enough so people can switch between Prometheus and other +similar tools. + +**b) go-kit metrics package as an interface** + +metrics package provides a set of uniform interfaces for service +instrumentation and offers adapters to popular metrics packages: + +https://godoc.org/github.com/go-kit/kit/metrics#pkg-subdirectories + +Comparing to Prometheus API, we're losing customisability and control, but gaining +freedom in choosing any instrument from the above list given we will extract +metrics creation into a separate function (see "providers" in node/node.go). + +**c) telegraf** + +Unlike already discussed options, telegraf does not require modifying Tendermint +source code. You create something called an input plugin, which polls +Tendermint RPC every second and calculates the metrics itself. + +While it may sound good, but some metrics we want to report are not exposed via +RPC or pubsub, therefore can't be accessed externally. + +**d) service, listening to pubsub** + +Same issue as the above. + +### 2. opencensus + +opencensus provides both metrics and tracing, which may be important in the +future. It's API looks different from go-kit and Prometheus, but looks like it +covers everything we need. + +Unfortunately, OpenCensus go client does not define any +interfaces, so if we want to abstract away metrics we +will need to write interfaces ourselves. + +### List of metrics + +| | Name | Type | Description | +| --- | ------------------------------------ | ------ | ----------------------------------------------------------------------------- | +| A | consensus_height | Gauge | | +| A | consensus_validators | Gauge | Number of validators who signed | +| A | consensus_validators_power | Gauge | Total voting power of all validators | +| A | consensus_missing_validators | Gauge | Number of validators who did not sign | +| A | consensus_missing_validators_power | Gauge | Total voting power of the missing validators | +| A | consensus_byzantine_validators | Gauge | Number of validators who tried to double sign | +| A | consensus_byzantine_validators_power | Gauge | Total voting power of the byzantine validators | +| A | consensus_block_interval | Timing | Time between this and last block (Block.Header.Time) | +| | consensus_block_time | Timing | Time to create a block (from creating a proposal to commit) | +| | consensus_time_between_blocks | Timing | Time between committing last block and (receiving proposal creating proposal) | +| A | consensus_rounds | Gauge | Number of rounds | +| | consensus_prevotes | Gauge | | +| | consensus_precommits | Gauge | | +| | consensus_prevotes_total_power | Gauge | | +| | consensus_precommits_total_power | Gauge | | +| A | consensus_num_txs | Gauge | | +| A | mempool_size | Gauge | | +| A | consensus_total_txs | Gauge | | +| A | consensus_block_size | Gauge | In bytes | +| A | p2p_peers | Gauge | Number of peers node's connected to | + +`A` - will be implemented in the fist place. + +**Proposed solution** + +## Status + +Implemented + +## Consequences + +### Positive + +Better visibility, support of variety of monitoring backends + +### Negative + +One more library to audit, messing metrics reporting code with business domain. + +### Neutral + +- diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-012-peer-transport.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-012-peer-transport.md new file mode 100644 index 00000000..1cf4fb80 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-012-peer-transport.md @@ -0,0 +1,113 @@ +# ADR 012: PeerTransport + +## Context + +One of the more apparent problems with the current architecture in the p2p +package is that there is no clear separation of concerns between different +components. Most notably the `Switch` is currently doing physical connection +handling. An artifact is the dependency of the Switch on +`[config.P2PConfig`](https://github.com/tendermint/tendermint/blob/05a76fb517f50da27b4bfcdc7b4cf185fc61eff6/config/config.go#L272-L339). + +Addresses: + +- [#2046](https://github.com/tendermint/tendermint/issues/2046) +- [#2047](https://github.com/tendermint/tendermint/issues/2047) + +First iteraton in [#2067](https://github.com/tendermint/tendermint/issues/2067) + +## Decision + +Transport concerns will be handled by a new component (`PeerTransport`) which +will provide Peers at its boundary to the caller. In turn `Switch` will use +this new component accept new `Peer`s and dial them based on `NetAddress`. + +### PeerTransport + +Responsible for emitting and connecting to Peers. The implementation of `Peer` +is left to the transport, which implies that the chosen transport dictates the +characteristics of the implementation handed back to the `Switch`. Each +transport implementation is responsible to filter establishing peers specific +to its domain, for the default multiplexed implementation the following will +apply: + +- connections from our own node +- handshake fails +- upgrade to secret connection fails +- prevent duplicate ip +- prevent duplicate id +- nodeinfo incompatibility + +```go +// PeerTransport proxies incoming and outgoing peer connections. +type PeerTransport interface { + // Accept returns a newly connected Peer. + Accept() (Peer, error) + + // Dial connects to a Peer. + Dial(NetAddress) (Peer, error) +} + +// EXAMPLE OF DEFAULT IMPLEMENTATION + +// multiplexTransport accepts tcp connections and upgrades to multiplexted +// peers. +type multiplexTransport struct { + listener net.Listener + + acceptc chan accept + closec <-chan struct{} + listenc <-chan struct{} + + dialTimeout time.Duration + handshakeTimeout time.Duration + nodeAddr NetAddress + nodeInfo NodeInfo + nodeKey NodeKey + + // TODO(xla): Remove when MConnection is refactored into mPeer. + mConfig conn.MConnConfig +} + +var _ PeerTransport = (*multiplexTransport)(nil) + +// NewMTransport returns network connected multiplexed peers. +func NewMTransport( + nodeAddr NetAddress, + nodeInfo NodeInfo, + nodeKey NodeKey, +) *multiplexTransport +``` + +### Switch + +From now the Switch will depend on a fully setup `PeerTransport` to +retrieve/reach out to its peers. As the more low-level concerns are pushed to +the transport, we can omit passing the `config.P2PConfig` to the Switch. + +```go +func NewSwitch(transport PeerTransport, opts ...SwitchOption) *Switch +``` + +## Status + +In Review. + +## Consequences + +### Positive + +- free Switch from transport concerns - simpler implementation +- pluggable transport implementation - simpler test setup +- remove Switch dependency on P2PConfig - easier to test + +### Negative + +- more setup for tests which depend on Switches + +### Neutral + +- multiplexed will be the default implementation + +[0] These guards could be potentially extended to be pluggable much like +middlewares to express different concerns required by differentally configured +environments. diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-013-symmetric-crypto.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-013-symmetric-crypto.md new file mode 100644 index 00000000..69bfc2f2 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-013-symmetric-crypto.md @@ -0,0 +1,99 @@ +# ADR 013: Need for symmetric cryptography + +## Context + +We require symmetric ciphers to handle how we encrypt keys in the sdk, +and to potentially encrypt `priv_validator.json` in tendermint. + +Currently we use AEAD's to support symmetric encryption, +which is great since we want data integrity in addition to privacy and authenticity. +We don't currently have a scenario where we want to encrypt without data integrity, +so it is fine to optimize our code to just use AEAD's. +Currently there is not a way to switch out AEAD's easily, this ADR outlines a way +to easily swap these out. + +### How do we encrypt with AEAD's + +AEAD's typically require a nonce in addition to the key. +For the purposes we require symmetric cryptography for, +we need encryption to be stateless. +Because of this we use random nonces. +(Thus the AEAD must support random nonces) + +We currently construct a random nonce, and encrypt the data with it. +The returned value is `nonce || encrypted data`. +The limitation of this is that does not provide a way to identify +which algorithm was used in encryption. +Consequently decryption with multiple algoritms is sub-optimal. +(You have to try them all) + +## Decision + +We should create the following two methods in a new `crypto/encoding/symmetric` package: + +```golang +func Encrypt(aead cipher.AEAD, plaintext []byte) (ciphertext []byte, err error) +func Decrypt(key []byte, ciphertext []byte) (plaintext []byte, err error) +func Register(aead cipher.AEAD, algo_name string, NewAead func(key []byte) (cipher.Aead, error)) error +``` + +This allows you to specify the algorithm in encryption, but not have to specify +it in decryption. +This is intended for ease of use in downstream applications, in addition to people +looking at the file directly. +One downside is that for the encrypt function you must have already initialized an AEAD, +but I don't really see this as an issue. + +If there is no error in encryption, Encrypt will return `algo_name || nonce || aead_ciphertext`. +`algo_name` should be length prefixed, using standard varuint encoding. +This will be binary data, but thats not a problem considering the nonce and ciphertext are also binary. + +This solution requires a mapping from aead type to name. +We can achieve this via reflection. + +```golang +func getType(myvar interface{}) string { + if t := reflect.TypeOf(myvar); t.Kind() == reflect.Ptr { + return "*" + t.Elem().Name() + } else { + return t.Name() + } +} +``` + +Then we maintain a map from the name returned from `getType(aead)` to `algo_name`. + +In decryption, we read the `algo_name`, and then instantiate a new AEAD with the key. +Then we call the AEAD's decrypt method on the provided nonce/ciphertext. + +`Register` allows a downstream user to add their own desired AEAD to the symmetric package. +It will error if the AEAD name is already registered. +This prevents a malicious import from modifying / nullifying an AEAD at runtime. + +## Implementation strategy + +The golang implementation of what is proposed is rather straight forward. +The concern is that we will break existing private keys if we just switch to this. +If this is concerning, we can make a simple script which doesn't require decoding privkeys, +for converting from the old format to the new one. + +## Status + +Proposed. + +## Consequences + +### Positive + +- Allows us to support new AEAD's, in a way that makes decryption easier +- Allows downstream users to add their own AEAD + +### Negative + +- We will have to break all private keys stored on disk. + They can be recovered using seed words, and upgrade scripts are simple. + +### Neutral + +- Caller has to instantiate the AEAD with the private key. + However it forces them to be aware of what signing algorithm they are using, which is a positive. diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-014-secp-malleability.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-014-secp-malleability.md new file mode 100644 index 00000000..33f9d004 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-014-secp-malleability.md @@ -0,0 +1,63 @@ +# ADR 014: Secp256k1 Signature Malleability + +## Context + +Secp256k1 has two layers of malleability. +The signer has a random nonce, and thus can produce many different valid signatures. +This ADR is not concerned with that. +The second layer of malleability basically allows one who is given a signature +to produce exactly one more valid signature for the same message from the same public key. +(They don't even have to know the message!) +The math behind this will be explained in the subsequent section. + +Note that in many downstream applications, signatures will appear in a transaction, and therefore in the tx hash. +This means that if someone broadcasts a transaction with secp256k1 signature, the signature can be altered into the other form by anyone in the p2p network. +Thus the tx hash will change, and this altered tx hash may be committed instead. +This breaks the assumption that you can broadcast a valid transaction and just wait for its hash to be included on chain. +One example is if you are broadcasting a tx in cosmos, +and you wait for it to appear on chain before incrementing your sequence number. +You may never increment your sequence number if a different tx hash got committed. +Removing this second layer of signature malleability concerns could ease downstream development. + +### ECDSA context + +Secp256k1 is ECDSA over a particular curve. +The signature is of the form `(r, s)`, where `s` is a field element. +(The particular field is the `Z_n`, where the elliptic curve has order `n`) +However `(r, -s)` is also another valid solution. +Note that anyone can negate a group element, and therefore can get this second signature. + +## Decision + +We can just distinguish a canonical form for the ECDSA signatures. +Then we require that all ECDSA signatures be in the form which we defined as canonical. +We reject signatures in non-canonical form. + +A canonical form is rather easy to define and check. +It would just be the smaller of the two values for `s`, defined lexicographically. +This is a simple check, instead of checking if `s < n`, instead check `s <= (n - 1)/2`. +An example of another cryptosystem using this +is the parity definition here https://github.com/zkcrypto/pairing/pull/30#issuecomment-372910663. + +This is the same solution Ethereum has chosen for solving secp malleability. + +## Proposed Implementation + +Fork https://github.com/btcsuite/btcd, and just update the [parse sig method](https://github.com/btcsuite/btcd/blob/11fcd83963ab0ecd1b84b429b1efc1d2cdc6d5c5/btcec/signature.go#L195) and serialize functions to enforce our canonical form. + +## Status + +Implemented + +## Consequences + +### Positive + +- Lets us maintain the ability to expect a tx hash to appear in the blockchain. + +### Negative + +- More work in all future implementations (Though this is a very simple check) +- Requires us to maintain another fork + +### Neutral diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-015-crypto-encoding.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-015-crypto-encoding.md new file mode 100644 index 00000000..bb0a8cd8 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-015-crypto-encoding.md @@ -0,0 +1,84 @@ +# ADR 015: Crypto encoding + +## Context + +We must standardize our method for encoding public keys and signatures on chain. +Currently we amino encode the public keys and signatures. +The reason we are using amino here is primarily due to ease of support in +parsing for other languages. +We don't need its upgradability properties in cryptosystems, as a change in +the crypto that requires adapting the encoding, likely warrants being deemed +a new cryptosystem. +(I.e. using new public parameters) + +## Decision + +### Public keys + +For public keys, we will continue to use amino encoding on the canonical +representation of the pubkey. +(Canonical as defined by the cryptosystem itself) +This has two significant drawbacks. +Amino encoding is less space-efficient, due to requiring support for upgradability. +Amino encoding support requires forking protobuf and adding this new interface support +option in the language of choice. + +The reason for continuing to use amino however is that people can create code +more easily in languages that already have an up to date amino library. +It is possible that this will change in the future, if it is deemed that +requiring amino for interacting with Tendermint cryptography is unnecessary. + +The arguments for space efficiency here are refuted on the basis that there are +far more egregious wastages of space in the SDK. +The space requirement of the public keys doesn't cause many problems beyond +increasing the space attached to each validator / account. + +The alternative to using amino here would be for us to create an enum type. +Switching to just an enum type is worthy of investigation post-launch. +For reference, part of amino encoding interfaces is basically a 4 byte enum +type definition. +Enum types would just change that 4 bytes to be a variant, and it would remove +the protobuf overhead, but it would be hard to integrate into the existing API. + +### Signatures + +Signatures should be switched to be `[]byte`. +Spatial efficiency in the signatures is quite important, +as it directly affects the gas cost of every transaction, +and the throughput of the chain. +Signatures don't need to encode what type they are for (unlike public keys) +since public keys must already be known. +Therefore we can validate the signature without needing to encode its type. + +When placed in state, signatures will still be amino encoded, but it will be the +primitive type `[]byte` getting encoded. + +#### Ed25519 + +Use the canonical representation for signatures. + +#### Secp256k1 + +There isn't a clear canonical representation here. +Signatures have two elements `r,s`. +These bytes are encoded as `r || s`, where `r` and `s` are both exactly +32 bytes long, encoded big-endian. +This is basically Ethereum's encoding, but without the leading recovery bit. + +## Status + +Implemented + +## Consequences + +### Positive + +- More space efficient signatures + +### Negative + +- We have an amino dependency for cryptography. + +### Neutral + +- No change to public keys diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-016-protocol-versions.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-016-protocol-versions.md new file mode 100644 index 00000000..e52743e5 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-016-protocol-versions.md @@ -0,0 +1,308 @@ +# ADR 016: Protocol Versions + +## TODO + +- How to / should we version the authenticated encryption handshake itself (ie. + upfront protocol negotiation for the P2PVersion) +- How to / should we version ABCI itself? Should it just be absorbed by the + BlockVersion? + +## Changelog + +- 18-09-2018: Updates after working a bit on implementation + - ABCI Handshake needs to happen independently of starting the app + conns so we can see the result + - Add question about ABCI protocol version +- 16-08-2018: Updates after discussion with SDK team + - Remove signalling for next version from Header/ABCI +- 03-08-2018: Updates from discussion with Jae: + - ProtocolVersion contains Block/AppVersion, not Current/Next + - signal upgrades to Tendermint using EndBlock fields + - dont restrict peer compatibilty by version to simplify syncing old nodes +- 28-07-2018: Updates from review + - split into two ADRs - one for protocol, one for chains + - include signalling for upgrades in header +- 16-07-2018: Initial draft - was originally joint ADR for protocol and chain + versions + +## Context + +Here we focus on software-agnostic protocol versions. + +The Software Version is covered by SemVer and described elsewhere. +It is not relevant to the protocol description, suffice to say that if any protocol version +changes, the software version changes, but not necessarily vice versa. + +Software version should be included in NodeInfo for convenience/diagnostics. + +We are also interested in versioning across different blockchains in a +meaningful way, for instance to differentiate branches of a contentious +hard-fork. We leave that for a later ADR. + +## Requirements + +We need to version components of the blockchain that may be independently upgraded. +We need to do it in a way that is scalable and maintainable - we can't just litter +the code with conditionals. + +We can consider the complete version of the protocol to contain the following sub-versions: +BlockVersion, P2PVersion, AppVersion. These versions reflect the major sub-components +of the software that are likely to evolve together, at different rates, and in different ways, +as described below. + +The BlockVersion defines the core of the blockchain data structures and +should change infrequently. + +The P2PVersion defines how peers connect and communicate with eachother - it's +not part of the blockchain data structures, but defines the protocols used to build the +blockchain. It may change gradually. + +The AppVersion determines how we compute app specific information, like the +AppHash and the Results. + +All of these versions may change over the life of a blockchain, and we need to +be able to help new nodes sync up across version changes. This means we must be willing +to connect to peers with older version. + +### BlockVersion + +- All tendermint hashed data-structures (headers, votes, txs, responses, etc.). + - Note the semantic meaning of a transaction may change according to the AppVersion, but the way txs are merklized into the header is part of the BlockVersion +- It should be the least frequent/likely to change. + - Tendermint should be stabilizing - it's just Atomic Broadcast. + - We can start considering for Tendermint v2.0 in a year +- It's easy to determine the version of a block from its serialized form + +### P2PVersion + +- All p2p and reactor messaging (messages, detectable behavior) +- Will change gradually as reactors evolve to improve performance and support new features - eg proposed new message types BatchTx in the mempool and HasBlockPart in the consensus +- It's easy to determine the version of a peer from its first serialized message/s +- New versions must be compatible with at least one old version to allow gradual upgrades + +### AppVersion + +- The ABCI state machine (txs, begin/endblock behavior, commit hashing) +- Behaviour and message types will change abruptly in the course of the life of a chain +- Need to minimize complexity of the code for supporting different AppVersions at different heights +- Ideally, each version of the software supports only a _single_ AppVersion at one time + - this means we checkout different versions of the software at different heights instead of littering the code + with conditionals + - minimize the number of data migrations required across AppVersion (ie. most AppVersion should be able to read the same state from disk as previous AppVersion). + +## Ideal + +Each component of the software is independently versioned in a modular way and its easy to mix and match and upgrade. + +## Proposal + +Each of BlockVersion, AppVersion, P2PVersion, is a monotonically increasing uint64. + +To use these versions, we need to update the block Header, the p2p NodeInfo, and the ABCI. + +### Header + +Block Header should include a `Version` struct as its first field like: + +``` +type Version struct { + Block uint64 + App uint64 +} +``` + +Here, `Version.Block` defines the rules for the current block, while +`Version.App` defines the app version that processed the last block and computed +the `AppHash` in the current block. Together they provide a complete description +of the consensus-critical protocol. + +Since we have settled on a proto3 header, the ability to read the BlockVersion out of the serialized header is unanimous. + +Using a Version struct gives us more flexibility to add fields without breaking +the header. + +The ProtocolVersion struct includes both the Block and App versions - it should +serve as a complete description of the consensus-critical protocol. + +### NodeInfo + +NodeInfo should include a Version struct as its first field like: + +``` +type Version struct { + P2P uint64 + Block uint64 + App uint64 + + Other []string +} +``` + +Note this effectively makes `Version.P2P` the first field in the NodeInfo, so it +should be easy to read this out of the serialized header if need be to facilitate an upgrade. + +The `Version.Other` here should include additional information like the name of the software client and +it's SemVer version - this is for convenience only. Eg. +`tendermint-core/v0.22.8`. It's a `[]string` so it can include information about +the version of Tendermint, of the app, of Tendermint libraries, etc. + +### ABCI + +Since the ABCI is responsible for keeping Tendermint and the App in sync, we +need to communicate version information through it. + +On startup, we use Info to perform a basic handshake. It should include all the +version information. + +We also need to be able to update versions in the life of a blockchain. The +natural place to do this is EndBlock. + +Note that currently the result of the Handshake isn't exposed anywhere, as the +handshaking happens inside the `proxy.AppConns` abstraction. We will need to +remove the handshaking from the `proxy` package so we can call it independently +and get the result, which should contain the application version. + +#### Info + +RequestInfo should add support for protocol versions like: + +``` +message RequestInfo { + string version + uint64 block_version + uint64 p2p_version +} +``` + +Similarly, ResponseInfo should return the versions: + +``` +message ResponseInfo { + string data + + string version + uint64 app_version + + int64 last_block_height + bytes last_block_app_hash +} +``` + +The existing `version` fields should be called `software_version` but we leave +them for now to reduce the number of breaking changes. + +#### EndBlock + +Updating the version could be done either with new fields or by using the +existing `tags`. Since we're trying to communicate information that will be +included in Tendermint block Headers, it should be native to the ABCI, and not +something embedded through some scheme in the tags. Thus, version updates should +be communicated through EndBlock. + +EndBlock already contains `ConsensusParams`. We can add version information to +the ConsensusParams as well: + +``` +message ConsensusParams { + + BlockSize block_size + EvidenceParams evidence_params + VersionParams version +} + +message VersionParams { + uint64 block_version + uint64 app_version +} +``` + +For now, the `block_version` will be ignored, as we do not allow block version +to be updated live. If the `app_version` is set, it signals that the app's +protocol version has changed, and the new `app_version` will be included in the +`Block.Header.Version.App` for the next block. + +### BlockVersion + +BlockVersion is included in both the Header and the NodeInfo. + +Changing BlockVersion should happen quite infrequently and ideally only for +critical upgrades. For now, it is not encoded in ABCI, though it's always +possible to use tags to signal an external process to co-ordinate an upgrade. + +Note Ethereum has not had to make an upgrade like this (everything has been at state machine level, AFAIK). + +### P2PVersion + +P2PVersion is not included in the block Header, just the NodeInfo. + +P2PVersion is the first field in the NodeInfo. NodeInfo is also proto3 so this is easy to read out. + +Note we need the peer/reactor protocols to take the versions of peers into account when sending messages: + +- don't send messages they don't understand +- don't send messages they don't expect + +Doing this will be specific to the upgrades being made. + +Note we also include the list of reactor channels in the NodeInfo and already don't send messages for channels the peer doesn't understand. +If upgrades always use new channels, this simplifies the development cost of backwards compatibility. + +Note NodeInfo is only exchanged after the authenticated encryption handshake to ensure that it's private. +Doing any version exchange before encrypting could be considered information leakage, though I'm not sure +how much that matters compared to being able to upgrade the protocol. + +XXX: if needed, can we change the meaning of the first byte of the first message to encode a handshake version? +this is the first byte of a 32-byte ed25519 pubkey. + +### AppVersion + +AppVersion is also included in the block Header and the NodeInfo. + +AppVersion essentially defines how the AppHash and LastResults are computed. + +### Peer Compatibility + +Restricting peer compatibility based on version is complicated by the need to +help old peers, possibly on older versions, sync the blockchain. + +We might be tempted to say that we only connect to peers with the same +AppVersion and BlockVersion (since these define the consensus critical +computations), and a select list of P2PVersions (ie. those compatible with +ours), but then we'd need to make accomodations for connecting to peers with the +right Block/AppVersion for the height they're on. + +For now, we will connect to peers with any version and restrict compatibility +solely based on the ChainID. We leave more restrictive rules on peer +compatibiltiy to a future proposal. + +### Future Changes + +It may be valuable to support an `/unsafe_stop?height=_` endpoint to tell Tendermint to shutdown at a given height. +This could be use by an external manager process that oversees upgrades by +checking out and installing new software versions and restarting the process. It +would subscribe to the relevant upgrade event (needs to be implemented) and call `/unsafe_stop` at +the correct height (of course only after getting approval from its user!) + +## Consequences + +### Positive + +- Make tendermint and application versions native to the ABCI to more clearly + communicate about them +- Distinguish clearly between protocol versions and software version to + facilitate implementations in other languages +- Versions included in key data structures in easy to discern way +- Allows proposers to signal for upgrades and apps to decide when to actually change the + version (and start signalling for a new version) + +### Neutral + +- Unclear how to version the initial P2P handshake itself +- Versions aren't being used (yet) to restrict peer compatibility +- Signalling for a new version happens through the proposer and must be + tallied/tracked in the app. + +### Negative + +- Adds more fields to the ABCI +- Implies that a single codebase must be able to handle multiple versions diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-017-chain-versions.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-017-chain-versions.md new file mode 100644 index 00000000..7113dbae --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-017-chain-versions.md @@ -0,0 +1,99 @@ +# ADR 017: Chain Versions + +## TODO + +- clarify how to handle slashing when ChainID changes + +## Changelog + +- 28-07-2018: Updates from review + - split into two ADRs - one for protocol, one for chains +- 16-07-2018: Initial draft - was originally joint ADR for protocol and chain + versions + +## Context + +Software and Protocol versions are covered in a separate ADR. + +Here we focus on chain versions. + +## Requirements + +We need to version blockchains across protocols, networks, forks, etc. +We need chain identifiers and descriptions so we can talk about a multitude of chains, +and especially the differences between them, in a meaningful way. + +### Networks + +We need to support many independent networks running the same version of the software, +even possibly starting from the same initial state. +They must have distinct identifiers so that peers know which one they are joining and so +validators and users can prevent replay attacks. + +Call this the `NetworkName` (note we currently call this `ChainID` in the software. In this +ADR, ChainID has a different meaning). +It represents both the application being run and the community or intention +of running it. + +Peers only connect to other peers with the same NetworkName. + +### Forks + +We need to support existing networks upgrading and forking, wherein they may do any of: + + - revert back to some height, continue with the same versions but new blocks + - arbitrarily mutate state at some height, continue with the same versions (eg. Dao Fork) + - change the AppVersion at some height + +Note because of Tendermint's voting power threshold rules, a chain can only be extended under the "original" rules and under the new rules +if 1/3 or more is double signing, which is expressly prohibited, and is supposed to result in their punishment on both chains. Since they can censor +the punishment, the chain is expected to be hardforked to remove the validators. Thus, if both branches are to continue after a fork, +they will each require a new identifier, and the old chain identifier will be retired (ie. only useful for syncing history, not for new blocks).. + +TODO: explain how to handle slashing when chain id changed! + +We need a consistent way to describe forks. + +## Proposal + +### ChainDescription + +ChainDescription is a complete immutable description of a blockchain. It takes the following form: + +``` +ChainDescription = ///// +``` + +Here, StateHash is the merkle root of the initial state, ValHash is the merkle root of the initial Tendermint validator set, +and ConsensusParamsHash is the merkle root of the initial Tendermint consensus parameters. + +The `genesis.json` file must contain enough information to compute this value. It need not contain the StateHash or ValHash itself, +but contain the state from which they can be computed with the given protocol versions. + +NOTE: consider splitting NetworkName into NetworkName and AppName - this allows +folks to independently use the same application for different networks (ie we +could imagine multiple communities of validators wanting to put up a Hub using +the same app but having a distinct network name. Arguably not needed if +differences will come via different initial state / validators). + +#### ChainID + +Define `ChainID = TMHASH(ChainDescriptor)`. It's the unique ID of a blockchain. + +It should be Bech32 encoded when handled by users, eg. with `cosmoschain` prefix. + +#### Forks and Uprades + +When a chain forks or upgrades but continues the same history, it takes a new ChainDescription as follows: + +``` +ChainDescription = /x// +``` + +Where + +- ChainID is the ChainID from the previous ChainDescription (ie. its hash) +- `x` denotes that a change occured +- `Height` is the height the change occured +- ForkDescription has the same form as ChainDescription but for the fork +- this allows forks to specify new versions for tendermint or the app, as well as arbitrary changes to the state or validator set diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-018-ABCI-Validators.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-018-ABCI-Validators.md new file mode 100644 index 00000000..b517c369 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-018-ABCI-Validators.md @@ -0,0 +1,100 @@ +# ADR 018: ABCI Validator Improvements + +## Changelog + +016-08-2018: Follow up from review: - Revert changes to commit round - Remind about justification for removing pubkey - Update pros/cons +05-08-2018: Initial draft + +## Context + +ADR 009 introduced major improvements to the ABCI around validators and the use +of Amino. Here we follow up with some additional changes to improve the naming +and expected use of Validator messages. + +## Decision + +### Validator + +Currently a Validator contains `address` and `pub_key`, and one or the other is +optional/not-sent depending on the use case. Instead, we should have a +`Validator` (with just the address, used for RequestBeginBlock) +and a `ValidatorUpdate` (with the pubkey, used for ResponseEndBlock): + +``` +message Validator { + bytes address + int64 power +} + +message ValidatorUpdate { + PubKey pub_key + int64 power +} +``` + +As noted in [ADR-009](adr-009-ABCI-design.md), +the `Validator` does not contain a pubkey because quantum public keys are +quite large and it would be wasteful to send them all over ABCI with every block. +Thus, applications that want to take advantage of the information in BeginBlock +are _required_ to store pubkeys in state (or use much less efficient lazy means +of verifying BeginBlock data). + +### RequestBeginBlock + +LastCommitInfo currently has an array of `SigningValidator` that contains +information for each validator in the entire validator set. +Instead, this should be called `VoteInfo`, since it is information about the +validator votes. + +Note that all votes in a commit must be from the same round. + +``` +message LastCommitInfo { + int64 round + repeated VoteInfo commit_votes +} + +message VoteInfo { + Validator validator + bool signed_last_block +} +``` + +### ResponseEndBlock + +Use ValidatorUpdates instead of Validators. Then it's clear we don't need an +address, and we do need a pubkey. + +We could require the address here as well as a sanity check, but it doesn't seem +necessary. + +### InitChain + +Use ValidatorUpdates for both Request and Response. InitChain +is about setting/updating the initial validator set, unlike BeginBlock +which is just informational. + +## Status + +Implemented + +## Consequences + +### Positive + +- Clarifies the distinction between the different uses of validator information + +### Negative + +- Apps must still store the public keys in state to utilize the RequestBeginBlock info + +### Neutral + +- ResponseEndBlock does not require an address + +## References + +- [Latest ABCI Spec](https://github.com/tendermint/tendermint/blob/v0.22.8/docs/app-dev/abci-spec.md) +- [ADR-009](https://github.com/tendermint/tendermint/blob/v0.22.8/docs/architecture/adr-009-ABCI-design.md) +- [Issue #1712 - Don't send PubKey in + RequestBeginBlock](https://github.com/tendermint/tendermint/issues/1712) diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-019-multisigs.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-019-multisigs.md new file mode 100644 index 00000000..7fd3aab0 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-019-multisigs.md @@ -0,0 +1,162 @@ +# ADR 019: Encoding standard for Multisignatures + +## Changelog + +06-08-2018: Minor updates + +27-07-2018: Update draft to use amino encoding + +11-07-2018: Initial Draft + +5-26-2021: Multisigs were moved into the Cosmos-sdk + +## Context + +Multisignatures, or technically _Accountable Subgroup Multisignatures_ (ASM), +are signature schemes which enable any subgroup of a set of signers to sign any message, +and reveal to the verifier exactly who the signers were. +This allows for complex conditionals of when to validate a signature. + +Suppose the set of signers is of size _n_. +If we validate a signature if any subgroup of size _k_ signs a message, +this becomes what is commonly reffered to as a _k of n multisig_ in Bitcoin. + +This ADR specifies the encoding standard for general accountable subgroup multisignatures, +k of n accountable subgroup multisignatures, and its weighted variant. + +In the future, we can also allow for more complex conditionals on the accountable subgroup. + +## Proposed Solution + +### New structs + +Every ASM will then have its own struct, implementing the crypto.Pubkey interface. + +This ADR assumes that [replacing crypto.Signature with []bytes](https://github.com/tendermint/tendermint/issues/1957) has been accepted. + +#### K of N threshold signature + +The pubkey is the following struct: + +```golang +type ThresholdMultiSignaturePubKey struct { // K of N threshold multisig + K uint `json:"threshold"` + Pubkeys []crypto.Pubkey `json:"pubkeys"` +} +``` + +We will derive N from the length of pubkeys. (For spatial efficiency in encoding) + +`Verify` will expect an `[]byte` encoded version of the Multisignature. +(Multisignature is described in the next section) +The multisignature will be rejected if the bitmap has less than k indices, +or if any signature at any of the k indices is not a valid signature from +the kth public key on the message. +(If more than k signatures are included, all must be valid) + +`Bytes` will be the amino encoded version of the pubkey. + +Address will be `Hash(amino_encoded_pubkey)` + +The reason this doesn't use `log_8(n)` bytes per signer is because that heavily optimizes for the case where a very small number of signers are required. +e.g. for `n` of size `24`, that would only be more space efficient for `k < 3`. +This seems less likely, and that it should not be the case optimized for. + +#### Weighted threshold signature + +The pubkey is the following struct: + +```golang +type WeightedThresholdMultiSignaturePubKey struct { + Weights []uint `json:"weights"` + Threshold uint `json:"threshold"` + Pubkeys []crypto.Pubkey `json:"pubkeys"` +} +``` + +Weights and Pubkeys must be of the same length. +Everything else proceeds identically to the K of N multisig, +except the multisig fails if the sum of the weights is less than the threshold. + +#### Multisignature + +The inter-mediate phase of the signatures (as it accrues more signatures) will be the following struct: + +```golang +type Multisignature struct { + BitArray CryptoBitArray // Documented later + Sigs [][]byte +``` + +It is important to recall that each private key will output a signature on the provided message itself. +So no signing algorithm ever outputs the multisignature. +The UI will take a signature, cast into a multisignature, and then keep adding +new signatures into it, and when done marshal into `[]byte`. +This will require the following helper methods: + +```golang +func SigToMultisig(sig []byte, n int) +func GetIndex(pk crypto.Pubkey, []crypto.Pubkey) +func AddSignature(sig Signature, index int, multiSig *Multisignature) +``` + +The multisignature will be converted to an `[]byte` using amino.MarshalBinaryBare. \* + +#### Bit Array + +We would be using a new implementation of a bitarray. The struct it would be encoded/decoded from is + +```golang +type CryptoBitArray struct { + ExtraBitsStored byte `json:"extra_bits"` // The number of extra bits in elems. + Elems []byte `json:"elems"` +} +``` + +The reason for not using the BitArray currently implemented in `libs/common/bit_array.go` +is that it is less space efficient, due to a space / time trade-off. +Evidence for this is outlined in [this issue](https://github.com/tendermint/tendermint/issues/2077). + +In the multisig, we will not be performing arithmetic operations, +so there is no performance increase with the current implementation, +and just loss of spatial efficiency. +Implementing this new bit array with `[]byte` _should_ be simple, as no +arithmetic operations between bit arrays are required, and save a couple of bytes. +(Explained in that same issue) + +When this bit array encoded, the number of elements is encoded due to amino. +However we may be encoding a full byte for what we actually only need 1-7 bits for. +We store that difference in ExtraBitsStored. +This allows for us to have an unbounded number of signers, and is more space efficient than what is currently used in `libs/common`. +Again the implementation of this space saving feature is straight forward. + +### Encoding the structs + +We will use straight forward amino encoding. This is chosen for ease of compatibility in other languages. + +### Future points of discussion + +If desired, we can use ed25519 batch verification for all ed25519 keys. +This is a future point of discussion, but would be backwards compatible as this information won't need to be marshalled. +(There may even be cofactor concerns without ristretto) +Aggregation of pubkeys / sigs in Schnorr sigs / BLS sigs is not backwards compatible, and would need to be a new ASM type. + +## Status + +Implemented (moved to cosmos-sdk) + +## Consequences + +### Positive + +- Supports multisignatures, in a way that won't require any special cases in our downstream verification code. +- Easy to serialize / deserialize +- Unbounded number of signers + +### Negative + +- Larger codebase, however this should reside in a subfolder of tendermint/crypto, as it provides no new interfaces. (Ref #https://github.com/tendermint/go-crypto/issues/136) +- Space inefficient due to utilization of amino encoding +- Suggested implementation requires a new struct for every ASM. + +### Neutral diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-020-block-size.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-020-block-size.md new file mode 100644 index 00000000..f32ed7ab --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-020-block-size.md @@ -0,0 +1,104 @@ +# ADR 020: Limiting txs size inside a block + +## Changelog + +13-08-2018: Initial Draft +15-08-2018: Second version after Dev's comments +28-08-2018: Third version after Ethan's comments +30-08-2018: AminoOverheadForBlock => MaxAminoOverheadForBlock +31-08-2018: Bounding evidence and chain ID +13-01-2019: Add section on MaxBytes vs MaxDataBytes + +## Context + +We currently use MaxTxs to reap txs from the mempool when proposing a block, +but enforce MaxBytes when unmarshaling a block, so we could easily propose a +block thats too large to be valid. + +We should just remove MaxTxs all together and stick with MaxBytes, and have a +`mempool.ReapMaxBytes`. + +But we can't just reap BlockSize.MaxBytes, since MaxBytes is for the entire block, +not for the txs inside the block. There's extra amino overhead + the actual +headers on top of the actual transactions + evidence + last commit. +We could also consider using a MaxDataBytes instead of or in addition to MaxBytes. + +## MaxBytes vs MaxDataBytes + +The [PR #3045](https://github.com/tendermint/tendermint/pull/3045) suggested +additional clarity/justification was necessary here, wither respect to the use +of MaxDataBytes in addition to, or instead of, MaxBytes. + +MaxBytes provides a clear limit on the total size of a block that requires no +additional calculation if you want to use it to bound resource usage, and there +has been considerable discussions about optimizing tendermint around 1MB blocks. +Regardless, we need some maximum on the size of a block so we can avoid +unmarshaling blocks that are too big during the consensus, and it seems more +straightforward to provide a single fixed number for this rather than a +computation of "MaxDataBytes + everything else you need to make room for +(signatures, evidence, header)". MaxBytes provides a simple bound so we can +always say "blocks are less than X MB". + +Having both MaxBytes and MaxDataBytes feels like unnecessary complexity. It's +not particularly surprising for MaxBytes to imply the maximum size of the +entire block (not just txs), one just has to know that a block includes header, +txs, evidence, votes. For more fine grained control over the txs included in the +block, there is the MaxGas. In practice, the MaxGas may be expected to do most of +the tx throttling, and the MaxBytes to just serve as an upper bound on the total +size. Applications can use MaxGas as a MaxDataBytes by just taking the gas for +every tx to be its size in bytes. + +## Proposed solution + +Therefore, we should + +1) Get rid of MaxTxs. +2) Rename MaxTxsBytes to MaxBytes. + +When we need to ReapMaxBytes from the mempool, we calculate the upper bound as follows: + +``` +ExactLastCommitBytes = {number of validators currently enabled} * {MaxVoteBytes} +MaxEvidenceBytesPerBlock = MaxBytes / 10 +ExactEvidenceBytes = cs.evpool.PendingEvidence(MaxEvidenceBytesPerBlock) * MaxEvidenceBytes + +mempool.ReapMaxBytes(MaxBytes - MaxAminoOverheadForBlock - ExactLastCommitBytes - ExactEvidenceBytes - MaxHeaderBytes) +``` + +where MaxVoteBytes, MaxEvidenceBytes, MaxHeaderBytes and MaxAminoOverheadForBlock +are constants defined inside the `types` package: + +- MaxVoteBytes - 170 bytes +- MaxEvidenceBytes - 364 bytes +- MaxHeaderBytes - 476 bytes (~276 bytes hashes + 200 bytes - 50 UTF-8 encoded + symbols of chain ID 4 bytes each in the worst case + amino overhead) +- MaxAminoOverheadForBlock - 8 bytes (assuming MaxHeaderBytes includes amino + overhead for encoding header, MaxVoteBytes - for encoding vote, etc.) + +ChainID needs to bound to 50 symbols max. + +When reaping evidence, we use MaxBytes to calculate the upper bound (e.g. 1/10) +to save some space for transactions. + +NOTE while reaping the `max int` bytes in mempool, we should account that every +transaction will take `len(tx)+aminoOverhead`, where aminoOverhead=1-4 bytes. + +We should write a test that fails if the underlying structs got changed, but +MaxXXX stayed the same. + +## Status + +Implemented + +## Consequences + +### Positive + +* one way to limit the size of a block +* less variables to configure + +### Negative + +* constants that need to be adjusted if the underlying structs got changed + +### Neutral diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-021-abci-events.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-021-abci-events.md new file mode 100644 index 00000000..7cc062a4 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-021-abci-events.md @@ -0,0 +1,52 @@ +# ADR 012: ABCI Events + +## Changelog + +- *2018-09-02* Remove ABCI errors component. Update description for events +- *2018-07-12* Initial version + +## Context + +ABCI tags were first described in [ADR 002](https://github.com/tendermint/tendermint/blob/main/docs/architecture/adr-002-event-subscription.md). +They are key-value pairs that can be used to index transactions. + +Currently, ABCI messages return a list of tags to describe an +"event" that took place during the Check/DeliverTx/Begin/EndBlock, +where each tag refers to a different property of the event, like the sending and receiving account addresses. + +Since there is only one list of tags, recording data for multiple such events in +a single Check/DeliverTx/Begin/EndBlock must be done using prefixes in the key +space. + +Alternatively, groups of tags that constitute an event can be separated by a +special tag that denotes a break between the events. This would allow +straightforward encoding of multiple events into a single list of tags without +prefixing, at the cost of these "special" tags to separate the different events. + +TODO: brief description of how the indexing works + +## Decision + +Instead of returning a list of tags, return a list of events, where +each event is a list of tags. This way we naturally capture the concept of +multiple events happening during a single ABCI message. + +TODO: describe impact on indexing and querying + +## Status + +Implemented + +## Consequences + +### Positive + +- Ability to track distinct events separate from ABCI calls (DeliverTx/BeginBlock/EndBlock) +- More powerful query abilities + +### Negative + +- More complex query syntax +- More complex search implementation + +### Neutral diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-022-abci-errors.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-022-abci-errors.md new file mode 100644 index 00000000..ce2c56a2 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-022-abci-errors.md @@ -0,0 +1,63 @@ +# ADR 022: ABCI Errors + +## Changelog + +- *2018-09-01* Initial version + +## Context + +ABCI errors should provide an abstraction between application details +and the client interface responsible for formatting & displaying errors to the user. + +Currently, this abstraction consists of a single integer (the `code`), where any +`code > 0` is considered an error (ie. invalid transaction) and all type +information about the error is contained in the code. This integer is +expected to be decoded by the client into a known error string, where any +more specific data is contained in the `data`. + +In a [previous conversation](https://github.com/tendermint/abci/issues/165#issuecomment-353704015), +it was suggested that not all non-zero codes need to be errors, hence why it's called `code` and not `error code`. +It is unclear exactly how the semantics of the `code` field will evolve, though +better lite-client proofs (like discussed for tags +[here](https://github.com/tendermint/tendermint/issues/1007#issuecomment-413917763)) +may play a role. + +Note that having all type information in a single integer +precludes an easy coordination method between "module implementers" and "client +implementers", especially for apps with many "modules". With an unbounded error domain (such as a string), module +implementers can pick a globally unique prefix & error code set, so client +implementers could easily implement support for "module A" regardless of which +particular blockchain network it was running in and which other modules were running with it. With +only error codes, globally unique codes are difficult/impossible, as the space +is finite and collisions are likely without an easy way to coordinate. + +For instance, while trying to build an ecosystem of modules that can be composed into a single +ABCI application, the Cosmos-SDK had to hack a higher level "codespace" into the +single integer so that each module could have its own space to express its +errors. + +## Decision + +Include a `string code_space` in all ABCI messages that have a `code`. +This allows applications to namespace the codes so they can experiment with +their own code schemes. + +It is the responsibility of applications to limit the size of the `code_space` +string. + +How the codespace is hashed into block headers (ie. so it can be queried +efficiently by lite clients) is left for a separate ADR. + +## Consequences + +## Positive + +- No need for complex codespacing on a single integer +- More expressive type system for errors + +## Negative + +- Another field in the response needs to be accounted for +- Some redundancy with `code` field +- May encourage more error/code type info to move to the `codespace` string, which + could impact lite clients. diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-023-ABCI-propose-tx.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-023-ABCI-propose-tx.md new file mode 100644 index 00000000..af8ea8b8 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-023-ABCI-propose-tx.md @@ -0,0 +1,183 @@ +# ADR 023: ABCI `ProposeTx` Method + +## Changelog + +25-06-2018: Initial draft based on [#1776](https://github.com/tendermint/tendermint/issues/1776) + +## Context + +[#1776](https://github.com/tendermint/tendermint/issues/1776) was +opened in relation to implementation of a Plasma child chain using Tendermint +Core as consensus/replication engine. + +Due to the requirements of [Minimal Viable Plasma (MVP)](https://ethresear.ch/t/minimal-viable-plasma/426) and [Plasma Cash](https://ethresear.ch/t/plasma-cash-plasma-with-much-less-per-user-data-checking/1298), it is necessary for ABCI apps to have a mechanism to handle the following cases (more may emerge in the near future): + +1. `deposit` transactions on the Root Chain, which must consist of a block + with a single transaction, where there are no inputs and only one output + made in favour of the depositor. In this case, a `block` consists of + a transaction with the following shape: + + ``` + [0, 0, 0, 0, #input1 - zeroed out + 0, 0, 0, 0, #input2 - zeroed out + , , #output1 - in favour of depositor + 0, 0, #output2 - zeroed out + , + ] + ``` + + `exit` transactions may also be treated in a similar manner, wherein the + input is the UTXO being exited on the Root Chain, and the output belongs to + a reserved "burn" address, e.g., `0x0`. In such cases, it is favorable for + the containing block to only hold a single transaction that may receive + special treatment. + +2. Other "internal" transactions on the child chain, which may be initiated + unilaterally. The most basic example of is a coinbase transaction + implementing validator node incentives, but may also be app-specific. In + these cases, it may be favorable for such transactions to + be ordered in a specific manner, e.g., coinbase transactions will always be + at index 0. In general, such strategies increase the determinism and + predictability of blockchain applications. + +While it is possible to deal with the cases enumerated above using the +existing ABCI, currently available result in suboptimal workarounds. Two are +explained in greater detail below. + +### Solution 1: App state-based Plasma chain + +In this work around, the app maintains a `PlasmaStore` with a corresponding +`Keeper`. The PlasmaStore is responsible for maintaing a second, separate +blockchain that complies with the MVP specification, including `deposit` +blocks and other "internal" transactions. These "virtual" blocks are then broadcasted +to the Root Chain. + +This naive approach is, however, fundamentally flawed, as it by definition +diverges from the canonical chain maintained by Tendermint. This is further +exacerbated if the business logic for generating such transactions is +potentially non-deterministic, as this should not even be done in +`Begin/EndBlock`, which may, as a result, break consensus guarantees. + +Additinoally, this has serious implications for "watchers" - independent third parties, +or even an auxilliary blockchain, responsible for ensuring that blocks recorded +on the Root Chain are consistent with the Plasma chain's. Since, in this case, +the Plasma chain is inconsistent with the canonical one maintained by Tendermint +Core, it seems that there exists no compact means of verifying the legitimacy of +the Plasma chain without replaying every state transition from genesis (!). + +### Solution 2: Broadcast to Tendermint Core from ABCI app + +This approach is inspired by `tendermint`, in which Ethereum transactions are +relayed to Tendermint Core. It requires the app to maintain a client connection +to the consensus engine. + +Whenever an "internal" transaction needs to be created, the proposer of the +current block broadcasts the transaction or transactions to Tendermint as +needed in order to ensure that the Tendermint chain and Plasma chain are +completely consistent. + +This allows "internal" transactions to pass through the full consensus +process, and can be validated in methods like `CheckTx`, i.e., signed by the +proposer, is the semantically correct, etc. Note that this involves informing +the ABCI app of the block proposer, which was temporarily hacked in as a means +of conducting this experiment, although this should not be necessary when the +current proposer is passed to `BeginBlock`. + +It is much easier to relay these transactions directly to the Root +Chain smart contract and/or maintain a "compressed" auxiliary chain comprised +of Plasma-friendly blocks that 100% reflect the canonical (Tendermint) +blockchain. Unfortunately, this approach not idiomatic (i.e., utilises the +Tendermint consensus engine in unintended ways). Additionally, it does not +allow the application developer to: + +- Control the _ordering_ of transactions in the proposed block (e.g., index 0, + or 0 to `n` for coinbase transactions) +- Control the _number_ of transactions in the block (e.g., when a `deposit` + block is required) + +Since determinism is of utmost importance in blockchain engineering, this approach, +while more viable, should also not be considered as fit for production. + +## Decision + +### `ProposeTx` + +In order to address the difficulties described above, the ABCI interface must +expose an additional method, tentatively named `ProposeTx`. + +It should have the following signature: + +``` +ProposeTx(RequestProposeTx) ResponseProposeTx +``` + +Where `RequestProposeTx` and `ResponseProposeTx` are `message`s with the +following shapes: + +``` +message RequestProposeTx { + int64 next_block_height = 1; // height of the block the proposed tx would be part of + Validator proposer = 2; // the proposer details +} + +message ResponseProposeTx { + int64 num_tx = 1; // the number of tx to include in proposed block + repeated bytes txs = 2; // ordered transaction data to include in block + bool exclusive = 3; // whether the block should include other transactions (from `mempool`) +} +``` + +`ProposeTx` would be called by before `mempool.Reap` at this +[line](https://github.com/tendermint/tendermint/blob/9cd9f3338bc80a12590631632c23c8dbe3ff5c34/consensus/state.go#L935). +Depending on whether `exclusive` is `true` or `false`, the proposed +transactions are then pushed on top of the transactions received from +`mempool.Reap`. + +### `DeliverTx` + +Since the list of `tx` received from `ProposeTx` are _not_ passed through `CheckTx`, +it is probably a good idea to provide a means of differentiatiating "internal" transactions +from user-generated ones, in case the app developer needs/wants to take extra measures to +ensure validity of the proposed transactions. + +Therefore, the `RequestDeliverTx` message should be changed to provide an additional flag, like so: + +``` +message RequestDeliverTx { + bytes tx = 1; + bool internal = 2; +} +``` + +Alternatively, an additional method `DeliverProposeTx` may be added as an accompanient to +`ProposeTx`. However, it is not clear at this stage if this additional overhead is necessary +to preserve consensus guarantees given that a simple flag may suffice for now. + +## Status + +Pending + +## Consequences + +### Positive + +- Tendermint ABCI apps will be able to function as minimally viable Plasma chains. +- It will thereby become possible to add an extension to `cosmos-sdk` to enable + ABCI apps to support both IBC and Plasma, maximising interop. +- ABCI apps will have great control and flexibility in managing blockchain state, + without having to resort to non-deterministic hacks and/or unsafe workarounds + +### Negative + +- Maintenance overhead of exposing additional ABCI method +- Potential security issues that may have been overlooked and must now be tested extensively + +### Neutral + +- ABCI developers must deal with increased (albeit nominal) API surface area. + +## References + +- [#1776 Plasma and "Internal" Transactions in ABCI Apps](https://github.com/tendermint/tendermint/issues/1776) +- [Minimal Viable Plasma](https://ethresear.ch/t/minimal-viable-plasma/426) +- [Plasma Cash: Plasma with much less per-user data checking](https://ethresear.ch/t/plasma-cash-plasma-with-much-less-per-user-data-checking/1298) diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-024-sign-bytes.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-024-sign-bytes.md new file mode 100644 index 00000000..ae19922a --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-024-sign-bytes.md @@ -0,0 +1,234 @@ +# ADR 024: SignBytes and validator types in privval + +## Context + +Currently, the messages exchanged between tendermint and a (potentially remote) signer/validator, +namely votes, proposals, and heartbeats, are encoded as a JSON string +(e.g., via `Vote.SignBytes(...)`) and then +signed . JSON encoding is sub-optimal for both, hardware wallets +and for usage in ethereum smart contracts. Both is laid down in detail in [issue#1622]. + +Also, there are currently no differences between sign-request and -replies. Also, there is no possibility +for a remote signer to include an error code or message in case something went wrong. +The messages exchanged between tendermint and a remote signer currently live in +[privval/socket.go] and encapsulate the corresponding types in [types]. + + +[privval/socket.go]: https://github.com/tendermint/tendermint/blob/d419fffe18531317c28c29a292ad7d253f6cafdf/privval/socket.go#L496-L502 +[issue#1622]: https://github.com/tendermint/tendermint/issues/1622 +[types]: https://github.com/tendermint/tendermint/tree/main/types + + +## Decision + +- restructure vote, proposal, and heartbeat such that their encoding is easily parseable by +hardware devices and smart contracts using a binary encoding format ([amino] in this case) +- split up the messages exchanged between tendermint and remote signers into requests and +responses (see details below) +- include an error type in responses + +### Overview +``` ++--------------+ +----------------+ +| | SignXRequest | | +|Remote signer |<---------------------+ tendermint | +| (e.g. KMS) | | | +| +--------------------->| | ++--------------+ SignedXReply +----------------+ + + +SignXRequest { + x: X +} + +SignedXReply { + x: X + sig: Signature // []byte + err: Error{ + code: int + desc: string + } +} +``` + +TODO: Alternatively, the type `X` might directly include the signature. A lot of places expect a vote with a +signature and do not necessarily deal with "Replies". +Still exploring what would work best here. +This would look like (exemplified using X = Vote): +``` +Vote { + // all fields besides signature +} + +SignedVote { + Vote Vote + Signature []byte +} + +SignVoteRequest { + Vote Vote +} + +SignedVoteReply { + Vote SignedVote + Err Error +} +``` + +**Note:** There was a related discussion around including a fingerprint of, or, the whole public-key +into each sign-request to tell the signer which corresponding private-key to +use to sign the message. This is particularly relevant in the context of the KMS +but is currently not considered in this ADR. + + +[amino]: https://github.com/tendermint/go-amino/ + +### Vote + +As explained in [issue#1622] `Vote` will be changed to contain the following fields +(notation in protobuf-like syntax for easy readability): + +```proto +// vanilla protobuf / amino encoded +message Vote { + Version fixed32 + Height sfixed64 + Round sfixed32 + VoteType fixed32 + Timestamp Timestamp // << using protobuf definition + BlockID BlockID // << as already defined + ChainID string // at the end because length could vary a lot +} + +// this is an amino registered type; like currently privval.SignVoteMsg: +// registered with "tendermint/socketpv/SignVoteRequest" +message SignVoteRequest { + Vote vote +} + +// amino registered type +// registered with "tendermint/socketpv/SignedVoteReply" +message SignedVoteReply { + Vote Vote + Signature Signature + Err Error +} + +// we will use this type everywhere below +message Error { + Type uint // error code + Description string // optional description +} + +``` + +The `ChainID` gets moved into the vote message directly. Previously, it was injected +using the [Signable] interface method `SignBytes(chainID string) []byte`. Also, the +signature won't be included directly, only in the corresponding `SignedVoteReply` message. + +[Signable]: https://github.com/tendermint/tendermint/blob/d419fffe18531317c28c29a292ad7d253f6cafdf/types/signable.go#L9-L11 + +### Proposal + +```proto +// vanilla protobuf / amino encoded +message Proposal { + Height sfixed64 + Round sfixed32 + Timestamp Timestamp // << using protobuf definition + BlockPartsHeader PartSetHeader // as already defined + POLRound sfixed32 + POLBlockID BlockID // << as already defined +} + +// amino registered with "tendermint/socketpv/SignProposalRequest" +message SignProposalRequest { + Proposal proposal +} + +// amino registered with "tendermint/socketpv/SignProposalReply" +message SignProposalReply { + Prop Proposal + Sig Signature + Err Error // as defined above +} +``` + +### Heartbeat + +**TODO**: clarify if heartbeat also needs a fixed offset and update the fields accordingly: + +```proto +message Heartbeat { + ValidatorAddress Address + ValidatorIndex int + Height int64 + Round int + Sequence int +} +// amino registered with "tendermint/socketpv/SignHeartbeatRequest" +message SignHeartbeatRequest { + Hb Heartbeat +} + +// amino registered with "tendermint/socketpv/SignHeartbeatReply" +message SignHeartbeatReply { + Hb Heartbeat + Sig Signature + Err Error // as defined above +} + +``` + +## PubKey + +TBA - this needs further thoughts: e.g. what todo like in the case of the KMS which holds +several keys? How does it know with which key to reply? + +## SignBytes +`SignBytes` will not require a `ChainID` parameter: + +```golang +type Signable interface { + SignBytes() []byte +} + +``` +And the implementation for vote, heartbeat, proposal will look like: +```golang +// type T is one of vote, sign, proposal +func (tp *T) SignBytes() []byte { + bz, err := cdc.MarshalBinary(tp) + if err != nil { + panic(err) + } + return bz +} +``` + +## Status + +Partially Accepted + +## Consequences + + + +### Positive + +The most relevant positive effect is that the signing bytes can easily be parsed by a +hardware module and a smart contract. Besides that: + +- clearer separation between requests and responses +- added error messages enable better error handling + + +### Negative + +- relatively huge change / refactoring touching quite some code +- lot's of places assume a `Vote` with a signature included -> they will need to +- need to modify some interfaces + +### Neutral + +not even the swiss are neutral diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-025-commit.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-025-commit.md new file mode 100644 index 00000000..a23d3803 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-025-commit.md @@ -0,0 +1,150 @@ +# ADR 025 Commit + +## Context + +Currently the `Commit` structure contains a lot of potentially redundant or unnecessary data. +It contains a list of precommits from every validator, where the precommit +includes the whole `Vote` structure. Thus each of the commit height, round, +type, and blockID are repeated for every validator, and could be deduplicated, +leading to very significant savings in block size. + +``` +type Commit struct { + BlockID BlockID `json:"block_id"` + Precommits []*Vote `json:"precommits"` +} + +type Vote struct { + ValidatorAddress Address `json:"validator_address"` + ValidatorIndex int `json:"validator_index"` + Height int64 `json:"height"` + Round int `json:"round"` + Timestamp time.Time `json:"timestamp"` + Type byte `json:"type"` + BlockID BlockID `json:"block_id"` + Signature []byte `json:"signature"` +} +``` + +The original tracking issue for this is [#1648](https://github.com/tendermint/tendermint/issues/1648). +We have discussed replacing the `Vote` type in `Commit` with a new `CommitSig` +type, which includes at minimum the vote signature. The `Vote` type will +continue to be used in the consensus reactor and elsewhere. + +A primary question is what should be included in the `CommitSig` beyond the +signature. One current constraint is that we must include a timestamp, since +this is how we calculuate BFT time, though we may be able to change this [in the +future](https://github.com/tendermint/tendermint/issues/2840). + +Other concerns here include: + +- Validator Address [#3596](https://github.com/tendermint/tendermint/issues/3596) - + Should the CommitSig include the validator address? It is very convenient to + do so, but likely not necessary. This was also discussed in [#2226](https://github.com/tendermint/tendermint/issues/2226). +- Absent Votes [#3591](https://github.com/tendermint/tendermint/issues/3591) - + How to represent absent votes? Currently they are just present as `nil` in the + Precommits list, which is actually problematic for serialization +- Other BlockIDs [#3485](https://github.com/tendermint/tendermint/issues/3485) - + How to represent votes for nil and for other block IDs? We currently allow + votes for nil and votes for alternative block ids, but just ignore them + + +## Decision + +Deduplicate the fields and introduce `CommitSig`: + +``` +type Commit struct { + Height int64 + Round int + BlockID BlockID `json:"block_id"` + Precommits []CommitSig `json:"precommits"` +} + +type CommitSig struct { + BlockID BlockIDFlag + ValidatorAddress Address + Timestamp time.Time + Signature []byte +} + + +// indicate which BlockID the signature is for +type BlockIDFlag int + +const ( + BlockIDFlagAbsent BlockIDFlag = iota // vote is not included in the Commit.Precommits + BlockIDFlagCommit // voted for the Commit.BlockID + BlockIDFlagNil // voted for nil +) + +``` + +Re the concerns outlined in the context: + +**Timestamp**: Leave the timestamp for now. Removing it and switching to +proposer based time will take more analysis and work, and will be left for a +future breaking change. In the meantime, the concerns with the current approach to +BFT time [can be +mitigated](https://github.com/tendermint/tendermint/issues/2840#issuecomment-529122431). + +**ValidatorAddress**: we include it in the `CommitSig` for now. While this +does increase the block size unecessarily (20-bytes per validator), it has some ergonomic and debugging advantages: + +- `Commit` contains everything necessary to reconstruct `[]Vote`, and doesn't depend on additional access to a `ValidatorSet` +- Lite clients can check if they know the validators in a commit without + re-downloading the validator set +- Easy to see directly in a commit which validators signed what without having + to fetch the validator set + +If and when we change the `CommitSig` again, for instance to remove the timestamp, +we can reconsider whether the ValidatorAddress should be removed. + +**Absent Votes**: we include absent votes explicitly with no Signature or +Timestamp but with the ValidatorAddress. This should resolve the serialization +issues and make it easy to see which validator's votes failed to be included. + +**Other BlockIDs**: We use a single byte to indicate which blockID a `CommitSig` +is for. The only options are: + - `Absent` - no vote received from the this validator, so no signature + - `Nil` - validator voted Nil - meaning they did not see a polka in time + - `Commit` - validator voted for this block + +Note this means we don't allow votes for any other blockIDs. If a signature is +included in a commit, it is either for nil or the correct blockID. According to +the Tendermint protocol and assumptions, there is no way for a correct validator to +precommit for a conflicting blockID in the same round an actual commit was +created. This was the consensus from +[#3485](https://github.com/tendermint/tendermint/issues/3485) + +We may want to consider supporting other blockIDs later, as a way to capture +evidence that might be helpful. We should clarify if/when/how doing so would +actually help first. To implement it, we could change the `Commit.BlockID` +field to a slice, where the first entry is the correct block ID and the other +entries are other BlockIDs that validators precommited before. The BlockIDFlag +enum can be extended to represent these additional block IDs on a per block +basis. + +## Status + +Implemented + +## Consequences + +### Positive + +Removing the Type/Height/Round/Index and the BlockID saves roughly 80 bytes per precommit. +It varies because some integers are varint. The BlockID contains two 32-byte hashes an integer, +and the Height is 8-bytes. + +For a chain with 100 validators, that's up to 8kB in savings per block! + + +### Negative + +- Large breaking change to the block and commit structure +- Requires differentiating in code between the Vote and CommitSig objects, which may add some complexity (votes need to be reconstructed to be verified and gossiped) + +### Neutral + +- Commit.Precommits no longer contains nil values diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-026-general-merkle-proof.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-026-general-merkle-proof.md new file mode 100644 index 00000000..5774c10f --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-026-general-merkle-proof.md @@ -0,0 +1,49 @@ +# ADR 026: General Merkle Proof + +## Context + +We are using raw `[]byte` for merkle proofs in `abci.ResponseQuery`. It makes hard to handle multilayer merkle proofs and general cases. Here, new interface `ProofOperator` is defined. The users can defines their own Merkle proof format and layer them easily. + +Goals: +- Layer Merkle proofs without decoding/reencoding +- Provide general way to chain proofs +- Make the proof format extensible, allowing thirdparty proof types + +## Decision + +### ProofOperator + +`type ProofOperator` is an interface for Merkle proofs. The definition is: + +```go +type ProofOperator interface { + Run([][]byte) ([][]byte, error) + GetKey() []byte + ProofOp() ProofOp +} +``` + +Since a proof can treat various data type, `Run()` takes `[][]byte` as the argument, not `[]byte`. For example, a range proof's `Run()` can take multiple key-values as its argument. It will then return the root of the tree for the further process, calculated with the input value. + +`ProofOperator` does not have to be a Merkle proof - it can be a function that transforms the argument for intermediate process e.g. prepending the length to the `[]byte`. + +### ProofOp + +`type ProofOp` is a protobuf message which is a triple of `Type string`, `Key []byte`, and `Data []byte`. `ProofOperator` and `ProofOp`are interconvertible, using `ProofOperator.ProofOp()` and `OpDecoder()`, where `OpDecoder` is a function that each proof type can register for their own encoding scheme. For example, we can add an byte for encoding scheme before the serialized proof, supporting JSON decoding. + +## Status + +Implemented + +## Consequences + +### Positive + +- Layering becomes easier (no encoding/decoding at each step) +- Thirdparty proof format is available + +### Negative + +- Larger size for abci.ResponseQuery +- Unintuitive proof chaining(it is not clear what `Run()` is doing) +- Additional codes for registering `OpDecoder`s diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-029-check-tx-consensus.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-029-check-tx-consensus.md new file mode 100644 index 00000000..191a0ec8 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-029-check-tx-consensus.md @@ -0,0 +1,127 @@ +# ADR 029: Check block txs before prevote + +## Changelog + +04-10-2018: Update with link to issue +[#2384](https://github.com/tendermint/tendermint/issues/2384) and reason for rejection +19-09-2018: Initial Draft + +## Context + +We currently check a tx's validity through 2 ways. + +1. Through checkTx in mempool connection. +2. Through deliverTx in consensus connection. + +The 1st is called when external tx comes in, so the node should be a proposer this time. The 2nd is called when external block comes in and reach the commit phase, the node doesn't need to be the proposer of the block, however it should check the txs in that block. + +In the 2nd situation, if there are many invalid txs in the block, it would be too late for all nodes to discover that most txs in the block are invalid, and we'd better not record invalid txs in the blockchain too. + +## Proposed solution + +Therefore, we should find a way to check the txs' validity before send out a prevote. Currently we have cs.isProposalComplete() to judge whether a block is complete. We can have + +``` +func (blockExec *BlockExecutor) CheckBlock(block *types.Block) error { + // check txs of block. + for _, tx := range block.Txs { + reqRes := blockExec.proxyApp.CheckTxAsync(tx) + reqRes.Wait() + if reqRes.Response == nil || reqRes.Response.GetCheckTx() == nil || reqRes.Response.GetCheckTx().Code != abci.CodeTypeOK { + return errors.Errorf("tx %v check failed. response: %v", tx, reqRes.Response) + } + } + return nil +} +``` + +such a method in BlockExecutor to check all txs' validity in that block. + +However, this method should not be implemented like that, because checkTx will share the same state used in mempool in the app. So we should define a new interface method checkBlock in Application to indicate it to use the same state as deliverTx. + +``` +type Application interface { + // Info/Query Connection + Info(RequestInfo) ResponseInfo // Return application info + Query(RequestQuery) ResponseQuery // Query for state + + // Mempool Connection + CheckTx(tx []byte) ResponseCheckTx // Validate a tx for the mempool + + // Consensus Connection + InitChain(RequestInitChain) ResponseInitChain // Initialize blockchain with validators and other info from TendermintCore + CheckBlock(RequestCheckBlock) ResponseCheckBlock + BeginBlock(RequestBeginBlock) ResponseBeginBlock // Signals the beginning of a block + DeliverTx(tx []byte) ResponseDeliverTx // Deliver a tx for full processing + EndBlock(RequestEndBlock) ResponseEndBlock // Signals the end of a block, returns changes to the validator set + Commit() ResponseCommit // Commit the state and return the application Merkle root hash +} +``` + +All app should implement that method. For example, counter: + +``` +func (app *CounterApplication) CheckBlock(block types.Request_CheckBlock) types.ResponseCheckBlock { + if app.serial { + app.originalTxCount = app.txCount //backup the txCount state + for _, tx := range block.CheckBlock.Block.Txs { + if len(tx) > 8 { + return types.ResponseCheckBlock{ + Code: code.CodeTypeEncodingError, + Log: fmt.Sprintf("Max tx size is 8 bytes, got %d", len(tx))} + } + tx8 := make([]byte, 8) + copy(tx8[len(tx8)-len(tx):], tx) + txValue := binary.BigEndian.Uint64(tx8) + if txValue < uint64(app.txCount) { + return types.ResponseCheckBlock{ + Code: code.CodeTypeBadNonce, + Log: fmt.Sprintf("Invalid nonce. Expected >= %v, got %v", app.txCount, txValue)} + } + app.txCount++ + } + } + return types.ResponseCheckBlock{Code: code.CodeTypeOK} +} +``` + +In BeginBlock, the app should restore the state to the orignal state before checking the block: + +``` +func (app *CounterApplication) DeliverTx(tx []byte) types.ResponseDeliverTx { + if app.serial { + app.txCount = app.originalTxCount //restore the txCount state + } + app.txCount++ + return types.ResponseDeliverTx{Code: code.CodeTypeOK} +} +``` + +The txCount is like the nonce in ethermint, it should be restored when entering the deliverTx phase. While some operation like checking the tx signature needs not to be done again. So the deliverTx can focus on how a tx can be applied, ignoring the checking of the tx, because all the checking has already been done in the checkBlock phase before. + +An optional optimization is alter the deliverTx to deliverBlock. For the block has already been checked by checkBlock, so all the txs in it are valid. So the app can cache the block, and in the deliverBlock phase, it just needs to apply the block in the cache. This optimization can save network current in deliverTx. + + + +## Status + +Rejected + +## Decision + +Performance impact is considered too great. See [#2384](https://github.com/tendermint/tendermint/issues/2384) + +## Consequences + +### Positive + +- more robust to defend the adversary to propose a block full of invalid txs. + +### Negative + +- add a new interface method. app logic needs to adjust to appeal to it. +- sending all the tx data over the ABCI twice +- potentially redundant validations (eg. signature checks in both CheckBlock and + DeliverTx) + +### Neutral diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-030-consensus-refactor.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-030-consensus-refactor.md new file mode 100644 index 00000000..5c8c3d75 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-030-consensus-refactor.md @@ -0,0 +1,458 @@ +# ADR 030: Consensus Refactor + +## Context + +One of the biggest challenges this project faces is to proof that the +implementations of the specifications are correct, much like we strive to +formaly verify our alogrithms and protocols we should work towards high +confidence about the correctness of our program code. One of those is the core +of Tendermint - Consensus - which currently resides in the `consensus` package. +Over time there has been high friction making changes to the package due to the +algorithm being scattered in a side-effectful container (the current +`ConsensusState`). In order to test the algorithm a large object-graph needs to +be set up and even than the non-deterministic parts of the container makes will +prevent high certainty. Where ideally we have a 1-to-1 representation of the +[spec](https://github.com/tendermint/spec), ready and easy to test for domain +experts. + +Addresses: + +- [#1495](https://github.com/tendermint/tendermint/issues/1495) +- [#1692](https://github.com/tendermint/tendermint/issues/1692) + +## Decision + +To remedy these issues we plan a gradual, non-invasive refactoring of the +`consensus` package. Starting of by isolating the consensus alogrithm into +a pure function and a finite state machine to address the most pressuring issue +of lack of confidence. Doing so while leaving the rest of the package in tact +and have follow-up optional changes to improve the sepration of concerns. + +### Implementation changes + +The core of Consensus can be modelled as a function with clear defined inputs: + +* `State` - data container for current round, height, etc. +* `Event`- significant events in the network + +producing clear outputs; + +* `State` - updated input +* `Message` - signal what actions to perform + +```go +type Event int + +const ( + EventUnknown Event = iota + EventProposal + Majority23PrevotesBlock + Majority23PrecommitBlock + Majority23PrevotesAny + Majority23PrecommitAny + TimeoutNewRound + TimeoutPropose + TimeoutPrevotes + TimeoutPrecommit +) + +type Message int + +const ( + MeesageUnknown Message = iota + MessageProposal + MessageVotes + MessageDecision +) + +type State struct { + height uint64 + round uint64 + step uint64 + lockedValue interface{} // TODO: Define proper type. + lockedRound interface{} // TODO: Define proper type. + validValue interface{} // TODO: Define proper type. + validRound interface{} // TODO: Define proper type. + // From the original notes: valid(v) + valid interface{} // TODO: Define proper type. + // From the original notes: proposer(h, r) + proposer interface{} // TODO: Define proper type. +} + +func Consensus(Event, State) (State, Message) { + // Consolidate implementation. +} +``` + +Tracking of relevant information to feed `Event` into the function and act on +the output is left to the `ConsensusExecutor` (formerly `ConsensusState`). + +Benefits for testing surfacing nicely as testing for a sequence of events +against algorithm could be as simple as the following example: + +``` go +func TestConsensusXXX(t *testing.T) { + type expected struct { + message Message + state State + } + + // Setup order of events, initial state and expectation. + var ( + events = []struct { + event Event + want expected + }{ + // ... + } + state = State{ + // ... + } + ) + + for _, e := range events { + sate, msg = Consensus(e.event, state) + + // Test message expectation. + if msg != e.want.message { + t.Fatalf("have %v, want %v", msg, e.want.message) + } + + // Test state expectation. + if !reflect.DeepEqual(state, e.want.state) { + t.Fatalf("have %v, want %v", state, e.want.state) + } + } +} +``` + + +## Consensus Executor + +## Consensus Core + +```go +type Event interface{} + +type EventNewHeight struct { + Height int64 + ValidatorId int +} + +type EventNewRound HeightAndRound + +type EventProposal struct { + Height int64 + Round int + Timestamp Time + BlockID BlockID + POLRound int + Sender int +} + +type Majority23PrevotesBlock struct { + Height int64 + Round int + BlockID BlockID +} + +type Majority23PrecommitBlock struct { + Height int64 + Round int + BlockID BlockID +} + +type HeightAndRound struct { + Height int64 + Round int +} + +type Majority23PrevotesAny HeightAndRound +type Majority23PrecommitAny HeightAndRound +type TimeoutPropose HeightAndRound +type TimeoutPrevotes HeightAndRound +type TimeoutPrecommit HeightAndRound + + +type Message interface{} + +type MessageProposal struct { + Height int64 + Round int + BlockID BlockID + POLRound int +} + +type VoteType int + +const ( + VoteTypeUnknown VoteType = iota + Prevote + Precommit +) + + +type MessageVote struct { + Height int64 + Round int + BlockID BlockID + Type VoteType +} + + +type MessageDecision struct { + Height int64 + Round int + BlockID BlockID +} + +type TriggerTimeout struct { + Height int64 + Round int + Duration Duration +} + + +type RoundStep int + +const ( + RoundStepUnknown RoundStep = iota + RoundStepPropose + RoundStepPrevote + RoundStepPrecommit + RoundStepCommit +) + +type State struct { + Height int64 + Round int + Step RoundStep + LockedValue BlockID + LockedRound int + ValidValue BlockID + ValidRound int + ValidatorId int + ValidatorSetSize int +} + +func proposer(height int64, round int) int {} +func getValue() BlockID {} + +func Consensus(event Event, state State) (State, Message, TriggerTimeout) { + msg = nil + timeout = nil + switch event := event.(type) { + case EventNewHeight: + if event.Height > state.Height { + state.Height = event.Height + state.Round = -1 + state.Step = RoundStepPropose + state.LockedValue = nil + state.LockedRound = -1 + state.ValidValue = nil + state.ValidRound = -1 + state.ValidatorId = event.ValidatorId + } + return state, msg, timeout + + case EventNewRound: + if event.Height == state.Height and event.Round > state.Round { + state.Round = eventRound + state.Step = RoundStepPropose + if proposer(state.Height, state.Round) == state.ValidatorId { + proposal = state.ValidValue + if proposal == nil { + proposal = getValue() + } + msg = MessageProposal { state.Height, state.Round, proposal, state.ValidRound } + } + timeout = TriggerTimeout { state.Height, state.Round, timeoutPropose(state.Round) } + } + return state, msg, timeout + + case EventProposal: + if event.Height == state.Height and event.Round == state.Round and + event.Sender == proposal(state.Height, state.Round) and state.Step == RoundStepPropose { + if event.POLRound >= state.LockedRound or event.BlockID == state.BlockID or state.LockedRound == -1 { + msg = MessageVote { state.Height, state.Round, event.BlockID, Prevote } + } + state.Step = RoundStepPrevote + } + return state, msg, timeout + + case TimeoutPropose: + if event.Height == state.Height and event.Round == state.Round and state.Step == RoundStepPropose { + msg = MessageVote { state.Height, state.Round, nil, Prevote } + state.Step = RoundStepPrevote + } + return state, msg, timeout + + case Majority23PrevotesBlock: + if event.Height == state.Height and event.Round == state.Round and state.Step >= RoundStepPrevote and event.Round > state.ValidRound { + state.ValidRound = event.Round + state.ValidValue = event.BlockID + if state.Step == RoundStepPrevote { + state.LockedRound = event.Round + state.LockedValue = event.BlockID + msg = MessageVote { state.Height, state.Round, event.BlockID, Precommit } + state.Step = RoundStepPrecommit + } + } + return state, msg, timeout + + case Majority23PrevotesAny: + if event.Height == state.Height and event.Round == state.Round and state.Step == RoundStepPrevote { + timeout = TriggerTimeout { state.Height, state.Round, timeoutPrevote(state.Round) } + } + return state, msg, timeout + + case TimeoutPrevote: + if event.Height == state.Height and event.Round == state.Round and state.Step == RoundStepPrevote { + msg = MessageVote { state.Height, state.Round, nil, Precommit } + state.Step = RoundStepPrecommit + } + return state, msg, timeout + + case Majority23PrecommitBlock: + if event.Height == state.Height { + state.Step = RoundStepCommit + state.LockedValue = event.BlockID + } + return state, msg, timeout + + case Majority23PrecommitAny: + if event.Height == state.Height and event.Round == state.Round { + timeout = TriggerTimeout { state.Height, state.Round, timeoutPrecommit(state.Round) } + } + return state, msg, timeout + + case TimeoutPrecommit: + if event.Height == state.Height and event.Round == state.Round { + state.Round = state.Round + 1 + } + return state, msg, timeout + } +} + +func ConsensusExecutor() { + proposal = nil + votes = HeightVoteSet { Height: 1 } + state = State { + Height: 1 + Round: 0 + Step: RoundStepPropose + LockedValue: nil + LockedRound: -1 + ValidValue: nil + ValidRound: -1 + } + + event = EventNewHeight {1, id} + state, msg, timeout = Consensus(event, state) + + event = EventNewRound {state.Height, 0} + state, msg, timeout = Consensus(event, state) + + if msg != nil { + send msg + } + + if timeout != nil { + trigger timeout + } + + for { + select { + case message := <- msgCh: + switch msg := message.(type) { + case MessageProposal: + + case MessageVote: + if msg.Height == state.Height { + newVote = votes.AddVote(msg) + if newVote { + switch msg.Type { + case Prevote: + prevotes = votes.Prevotes(msg.Round) + if prevotes.WeakCertificate() and msg.Round > state.Round { + event = EventNewRound { msg.Height, msg.Round } + state, msg, timeout = Consensus(event, state) + state = handleStateChange(state, msg, timeout) + } + + if blockID, ok = prevotes.TwoThirdsMajority(); ok and blockID != nil { + if msg.Round == state.Round and hasBlock(blockID) { + event = Majority23PrevotesBlock { msg.Height, msg.Round, blockID } + state, msg, timeout = Consensus(event, state) + state = handleStateChange(state, msg, timeout) + } + if proposal != nil and proposal.POLRound == msg.Round and hasBlock(blockID) { + event = EventProposal { + Height: state.Height + Round: state.Round + BlockID: blockID + POLRound: proposal.POLRound + Sender: message.Sender + } + state, msg, timeout = Consensus(event, state) + state = handleStateChange(state, msg, timeout) + } + } + + if prevotes.HasTwoThirdsAny() and msg.Round == state.Round { + event = Majority23PrevotesAny { msg.Height, msg.Round, blockID } + state, msg, timeout = Consensus(event, state) + state = handleStateChange(state, msg, timeout) + } + + case Precommit: + + } + } + } + case timeout := <- timeoutCh: + + case block := <- blockCh: + + } + } +} + +func handleStateChange(state, msg, timeout) State { + if state.Step == Commit { + state = ExecuteBlock(state.LockedValue) + } + if msg != nil { + send msg + } + if timeout != nil { + trigger timeout + } +} + +``` + +### Implementation roadmap + +* implement proposed implementation +* replace currently scattered calls in `ConsensusState` with calls to the new + `Consensus` function +* rename `ConsensusState` to `ConsensusExecutor` to avoid confusion +* propose design for improved separation and clear information flow between + `ConsensusExecutor` and `ConsensusReactor` + +## Status + +Draft. + +## Consequences + +### Positive + +- isolated implementation of the algorithm +- improved testability - simpler to proof correctness +- clearer separation of concerns - easier to reason + +### Negative + +### Neutral diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-033-pubsub.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-033-pubsub.md new file mode 100644 index 00000000..7b7912a9 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-033-pubsub.md @@ -0,0 +1,247 @@ +# ADR 033: pubsub 2.0 + +Author: Anton Kaliaev (@melekes) + +## Changelog + +02-10-2018: Initial draft + +16-01-2019: Second version based on our conversation with Jae + +17-01-2019: Third version explaining how new design solves current issues + +25-01-2019: Fourth version to treat buffered and unbuffered channels differently + +## Context + +Since the initial version of the pubsub, there's been a number of issues +raised: [#951], [#1879], [#1880]. Some of them are high-level issues questioning the +core design choices made. Others are minor and mostly about the interface of +`Subscribe()` / `Publish()` functions. + +### Sync vs Async + +Now, when publishing a message to subscribers, we can do it in a goroutine: + +_using channels for data transmission_ +```go +for each subscriber { + out := subscriber.outc + go func() { + out <- msg + } +} +``` + +_by invoking callback functions_ +```go +for each subscriber { + go subscriber.callbackFn() +} +``` + +This gives us greater performance and allows us to avoid "slow client problem" +(when other subscribers have to wait for a slow subscriber). A pool of +goroutines can be used to avoid uncontrolled memory growth. + +In certain cases, this is what you want. But in our case, because we need +strict ordering of events (if event A was published before B, the guaranteed +delivery order will be A -> B), we can't publish msg in a new goroutine every time. + +We can also have a goroutine per subscriber, although we'd need to be careful +with the number of subscribers. It's more difficult to implement as well + +unclear if we'll benefit from it (cause we'd be forced to create N additional +channels to distribute msg to these goroutines). + +### Non-blocking send + +There is also a question whenever we should have a non-blocking send. +Currently, sends are blocking, so publishing to one client can block on +publishing to another. This means a slow or unresponsive client can halt the +system. Instead, we can use a non-blocking send: + +```go +for each subscriber { + out := subscriber.outc + select { + case out <- msg: + default: + log("subscriber %v buffer is full, skipping...") + } +} +``` + +This fixes the "slow client problem", but there is no way for a slow client to +know if it had missed a message. We could return a second channel and close it +to indicate subscription termination. On the other hand, if we're going to +stick with blocking send, **devs must always ensure subscriber's handling code +does not block**, which is a hard task to put on their shoulders. + +The interim option is to run goroutines pool for a single message, wait for all +goroutines to finish. This will solve "slow client problem", but we'd still +have to wait `max(goroutine_X_time)` before we can publish the next message. + +### Channels vs Callbacks + +Yet another question is whether we should use channels for message transmission or +call subscriber-defined callback functions. Callback functions give subscribers +more flexibility - you can use mutexes in there, channels, spawn goroutines, +anything you really want. But they also carry local scope, which can result in +memory leaks and/or memory usage increase. + +Go channels are de-facto standard for carrying data between goroutines. + +### Why `Subscribe()` accepts an `out` channel? + +Because in our tests, we create buffered channels (cap: 1). Alternatively, we +can make capacity an argument and return a channel. + +## Decision + +### MsgAndTags + +Use a `MsgAndTags` struct on the subscription channel to indicate what tags the +msg matched. + +```go +type MsgAndTags struct { + Msg interface{} + Tags TagMap +} +``` + +### Subscription Struct + + +Change `Subscribe()` function to return a `Subscription` struct: + +```go +type Subscription struct { + // private fields +} + +func (s *Subscription) Out() <-chan MsgAndTags +func (s *Subscription) Canceled() <-chan struct{} +func (s *Subscription) Err() error +``` + +`Out()` returns a channel onto which messages and tags are published. +`Unsubscribe`/`UnsubscribeAll` does not close the channel to avoid clients from +receiving a nil message. + +`Canceled()` returns a channel that's closed when the subscription is terminated +and supposed to be used in a select statement. + +If the channel returned by `Canceled()` is not closed yet, `Err()` returns nil. +If the channel is closed, `Err()` returns a non-nil error explaining why: +`ErrUnsubscribed` if the subscriber choose to unsubscribe, +`ErrOutOfCapacity` if the subscriber is not pulling messages fast enough and the channel returned by `Out()` became full. +After `Err()` returns a non-nil error, successive calls to `Err() return the same error. + +```go +subscription, err := pubsub.Subscribe(...) +if err != nil { + // ... +} +for { +select { + case msgAndTags <- subscription.Out(): + // ... + case <-subscription.Canceled(): + return subscription.Err() +} +``` + +### Capacity and Subscriptions + +Make the `Out()` channel buffered (with capacity 1) by default. In most cases, we want to +terminate the slow subscriber. Only in rare cases, we want to block the pubsub +(e.g. when debugging consensus). This should lower the chances of the pubsub +being frozen. + +```go +// outCap can be used to set capacity of Out channel +// (1 by default, must be greater than 0). +Subscribe(ctx context.Context, clientID string, query Query, outCap... int) (Subscription, error) { +``` + +Use a different function for an unbuffered channel: + +```go +// Subscription uses an unbuffered channel. Publishing will block. +SubscribeUnbuffered(ctx context.Context, clientID string, query Query) (Subscription, error) { +``` + +SubscribeUnbuffered should not be exposed to users. + +### Blocking/Nonblocking + +The publisher should treat these kinds of channels separately. +It should block on unbuffered channels (for use with internal consensus events +in the consensus tests) and not block on the buffered ones. If a client is too +slow to keep up with it's messages, it's subscription is terminated: + +for each subscription { + out := subscription.outChan + if cap(out) == 0 { + // block on unbuffered channel + out <- msg + } else { + // don't block on buffered channels + select { + case out <- msg: + default: + // set the error, notify on the cancel chan + subscription.err = fmt.Errorf("client is too slow for msg) + close(subscription.cancelChan) + + // ... unsubscribe and close out + } + } +} + +### How this new design solves the current issues? + +[#951] ([#1880]): + +Because of non-blocking send, situation where we'll deadlock is not possible +anymore. If the client stops reading messages, it will be removed. + +[#1879]: + +MsgAndTags is used now instead of a plain message. + +### Future problems and their possible solutions + +[#2826] + +One question I am still pondering about: how to prevent pubsub from slowing +down consensus. We can increase the pubsub queue size (which is 0 now). Also, +it's probably a good idea to limit the total number of subscribers. + +This can be made automatically. Say we set queue size to 1000 and, when it's >= +80% full, refuse new subscriptions. + +## Status + +Implemented + +## Consequences + +### Positive + +- more idiomatic interface +- subscribers know what tags msg was published with +- subscribers aware of the reason their subscription was canceled + +### Negative + +- (since v1) no concurrency when it comes to publishing messages + +### Neutral + + +[#951]: https://github.com/tendermint/tendermint/issues/951 +[#1879]: https://github.com/tendermint/tendermint/issues/1879 +[#1880]: https://github.com/tendermint/tendermint/issues/1880 +[#2826]: https://github.com/tendermint/tendermint/issues/2826 diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-034-priv-validator-file-structure.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-034-priv-validator-file-structure.md new file mode 100644 index 00000000..c87cec13 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-034-priv-validator-file-structure.md @@ -0,0 +1,72 @@ +# ADR 034: PrivValidator file structure + +## Changelog + +03-11-2018: Initial Draft + +## Context + +For now, the PrivValidator file `priv_validator.json` contains mutable and immutable parts. +Even in an insecure mode which does not encrypt private key on disk, it is reasonable to separate +the mutable part and immutable part. + +References: +[#1181](https://github.com/tendermint/tendermint/issues/1181) +[#2657](https://github.com/tendermint/tendermint/issues/2657) +[#2313](https://github.com/tendermint/tendermint/issues/2313) + +## Proposed Solution + +We can split mutable and immutable parts with two structs: +```go +// FilePVKey stores the immutable part of PrivValidator +type FilePVKey struct { + Address types.Address `json:"address"` + PubKey crypto.PubKey `json:"pub_key"` + PrivKey crypto.PrivKey `json:"priv_key"` + + filePath string +} + +// FilePVState stores the mutable part of PrivValidator +type FilePVLastSignState struct { + Height int64 `json:"height"` + Round int `json:"round"` + Step int8 `json:"step"` + Signature []byte `json:"signature,omitempty"` + SignBytes cmn.HexBytes `json:"signbytes,omitempty"` + + filePath string + mtx sync.Mutex +} +``` + +Then we can combine `FilePVKey` with `FilePVLastSignState` and will get the original `FilePV`. + +```go +type FilePV struct { + Key FilePVKey + LastSignState FilePVLastSignState +} +``` + +As discussed, `FilePV` should be located in `config`, and `FilePVLastSignState` should be stored in `data`. The +store path of each file should be specified in `config.yml`. + +What we need to do next is changing the methods of `FilePV`. + +## Status + +Implemented + +## Consequences + +### Positive + +- separate the mutable and immutable of PrivValidator + +### Negative + +- need to add more config for file path + +### Neutral diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-035-documentation.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-035-documentation.md new file mode 100644 index 00000000..92cb0791 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-035-documentation.md @@ -0,0 +1,40 @@ +# ADR 035: Documentation + +Author: @zramsay (Zach Ramsay) + +## Changelog + +### November 2nd 2018 + +- initial write-up + +## Context + +The Tendermint documentation has undergone several changes until settling on the current model. Originally, the documentation was hosted on the website and had to be updated asynchronously from the code. Along with the other repositories requiring documentation, the whole stack moved to using Read The Docs to automatically generate, publish, and host the documentation. This, however, was insufficient; the RTD site had advertisement, it wasn't easily accessible to devs, didn't collect metrics, was another set of external links, etc. + +## Decision + +For two reasons, the decision was made to use VuePress: + +1) ability to get metrics (implemented on both Tendermint and SDK) +2) host the documentation on the website as a `/docs` endpoint. + +This is done while maintaining synchrony between the docs and code, i.e., the website is built whenever the docs are updated. + +## Status + +The two points above have been implemented; the `config.js` has a Google Analytics identifier and the documentation workflow has been up and running largely without problems for several months. Details about the documentation build & workflow can be found [here](../DOCS_README.md) + +## Consequences + +Because of the organizational seperation between Tendermint & Cosmos, there is a challenge of "what goes where" for certain aspects of documentation. + +### Positive + +This architecture is largely positive relative to prior docs arrangements. + +### Negative + +A significant portion of the docs automation / build process is in private repos with limited access/visibility to devs. However, these tasks are handled by the SRE team. + +### Neutral diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-036-empty-blocks-abci.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-036-empty-blocks-abci.md new file mode 100644 index 00000000..ec4806cf --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-036-empty-blocks-abci.md @@ -0,0 +1,38 @@ +# ADR 036: Empty Blocks via ABCI + +## Changelog + +- {date}: {changelog} + +## Context + +> This section contains all the context one needs to understand the current state, and why there is a problem. It should be as succinct as possible and introduce the high level idea behind the solution. + +## Decision + +> This section explains all of the details of the proposed solution, including implementation details. +> It should also describe affects / corollary items that may need to be changed as a part of this. +> If the proposed change will be large, please also indicate a way to do the change to maximize ease of review. +> (e.g. the optimal split of things to do between separate PR's) + +## Status + +> A decision may be "proposed" if it hasn't been agreed upon yet, or "accepted" once it is agreed upon. If a later ADR changes or reverses a decision, it may be marked as "deprecated" or "superseded" with a reference to its replacement. + +{Deprecated|Proposed|Accepted|Declined} + +## Consequences + +> This section describes the consequences, after applying the decision. All consequences should be summarized here, not just the "positive" ones. + +### Positive + +### Negative + +### Neutral + +## References + +> Are there any relevant PR comments, issues that led up to this, or articles referenced for why we made the given design choice? If so link them here! + +- {reference link} diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-037-deliver-block.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-037-deliver-block.md new file mode 100644 index 00000000..c5e119c0 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-037-deliver-block.md @@ -0,0 +1,100 @@ +# ADR 037: Deliver Block + +Author: Daniil Lashin (@danil-lashin) + +## Changelog + +13-03-2019: Initial draft + +## Context + +Initial conversation: https://github.com/tendermint/tendermint/issues/2901 + +Some applications can handle transactions in parallel, or at least some +part of tx processing can be parallelized. Now it is not possible for developer +to execute txs in parallel because Tendermint delivers them consequentially. + +## Decision + +Now Tendermint have `BeginBlock`, `EndBlock`, `Commit`, `DeliverTx` steps +while executing block. This doc proposes merging this steps into one `DeliverBlock` +step. It will allow developers of applications to decide how they want to +execute transactions (in parallel or consequentially). Also it will simplify and +speed up communications between application and Tendermint. + +As @jaekwon [mentioned](https://github.com/tendermint/tendermint/issues/2901#issuecomment-477746128) +in discussion not all application will benefit from this solution. In some cases, +when application handles transaction consequentially, it way slow down the blockchain, +because it need to wait until full block is transmitted to application to start +processing it. Also, in the case of complete change of ABCI, we need to force all the apps +to change their implementation completely. That's why I propose to introduce one more ABCI +type. + +# Implementation Changes + +In addition to default application interface which now have this structure + +```go +type Application interface { + // Info and Mempool methods... + + // Consensus Connection + InitChain(RequestInitChain) ResponseInitChain // Initialize blockchain with validators and other info from TendermintCore + BeginBlock(RequestBeginBlock) ResponseBeginBlock // Signals the beginning of a block + DeliverTx(tx []byte) ResponseDeliverTx // Deliver a tx for full processing + EndBlock(RequestEndBlock) ResponseEndBlock // Signals the end of a block, returns changes to the validator set + Commit() ResponseCommit // Commit the state and return the application Merkle root hash +} +``` + +this doc proposes to add one more: + +```go +type Application interface { + // Info and Mempool methods... + + // Consensus Connection + InitChain(RequestInitChain) ResponseInitChain // Initialize blockchain with validators and other info from TendermintCore + DeliverBlock(RequestDeliverBlock) ResponseDeliverBlock // Deliver full block + Commit() ResponseCommit // Commit the state and return the application Merkle root hash +} + +type RequestDeliverBlock struct { + Hash []byte + Header Header + Txs Txs + LastCommitInfo LastCommitInfo + ByzantineValidators []Evidence +} + +type ResponseDeliverBlock struct { + ValidatorUpdates []ValidatorUpdate + ConsensusParamUpdates *ConsensusParams + Tags []kv.Pair + TxResults []ResponseDeliverTx +} + +``` + +Also, we will need to add new config param, which will specify what kind of ABCI application uses. +For example, it can be `abci_type`. Then we will have 2 types: +- `advanced` - current ABCI +- `simple` - proposed implementation + +## Status + +In review + +## Consequences + +### Positive + +- much simpler introduction and tutorials for new developers (instead of implementing 5 methods whey +will need to implement only 3) +- txs can be handled in parallel +- simpler interface +- faster communications between Tendermint and application + +### Negative + +- Tendermint should now support 2 kinds of ABCI diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-038-non-zero-start-height.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-038-non-zero-start-height.md new file mode 100644 index 00000000..7dd474ec --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-038-non-zero-start-height.md @@ -0,0 +1,38 @@ +# ADR 038: Non-zero start height + +## Changelog + +- {date}: {changelog} + +## Context + +> This section contains all the context one needs to understand the current state, and why there is a problem. It should be as succinct as possible and introduce the high level idea behind the solution. + +## Decision + +> This section explains all of the details of the proposed solution, including implementation details. +> It should also describe affects / corollary items that may need to be changed as a part of this. +> If the proposed change will be large, please also indicate a way to do the change to maximize ease of review. +> (e.g. the optimal split of things to do between separate PR's) + +## Status + +> A decision may be "proposed" if it hasn't been agreed upon yet, or "accepted" once it is agreed upon. If a later ADR changes or reverses a decision, it may be marked as "deprecated" or "superseded" with a reference to its replacement. + +{Deprecated|Proposed|Accepted|Declined} + +## Consequences + +> This section describes the consequences, after applying the decision. All consequences should be summarized here, not just the "positive" ones. + +### Positive + +### Negative + +### Neutral + +## References + +> Are there any relevant PR comments, issues that led up to this, or articles referenced for why we made the given design choice? If so link them here! + +- {reference link} diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-039-peer-behaviour.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-039-peer-behaviour.md new file mode 100644 index 00000000..2fb63f4b --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-039-peer-behaviour.md @@ -0,0 +1,159 @@ +# ADR 039: Peer Behaviour Interface + +## Changelog +* 07-03-2019: Initial draft +* 14-03-2019: Updates from feedback + +## Context + +The responsibility for signaling and acting upon peer behavior lacks a single +owning component and is heavily coupled with the network stack[1](#references). Reactors +maintain a reference to the `p2p.Switch` which they use to call +`switch.StopPeerForError(...)` when a peer misbehaves and +`switch.MarkAsGood(...)` when a peer contributes in some meaningful way. +While the switch handles `StopPeerForError` internally, the `MarkAsGood` +method delegates to another component, `p2p.AddrBook`. This scheme of delegation +across Switch obscures the responsibility for handling peer behavior +and ties up the reactors in a larger dependency graph when testing. + +## Decision + +Introduce a `PeerBehaviour` interface and concrete implementations which +provide methods for reactors to signal peer behavior without direct +coupling `p2p.Switch`. Introduce a ErrorBehaviourPeer to provide +concrete reasons for stopping peers. Introduce GoodBehaviourPeer to provide +concrete ways in which a peer contributes. + +### Implementation Changes + +PeerBehaviour then becomes an interface for signaling peer errors as well +as for marking peers as `good`. + +```go +type PeerBehaviour interface { + Behaved(peer Peer, reason GoodBehaviourPeer) + Errored(peer Peer, reason ErrorBehaviourPeer) +} +``` + +Instead of signaling peers to stop with arbitrary reasons: +`reason interface{}` + +We introduce a concrete error type ErrorBehaviourPeer: +```go +type ErrorBehaviourPeer int + +const ( + ErrorBehaviourUnknown = iota + ErrorBehaviourBadMessage + ErrorBehaviourMessageOutofOrder + ... +) +``` + +To provide additional information on the ways a peer contributed, we introduce +the GoodBehaviourPeer type. + +```go +type GoodBehaviourPeer int + +const ( + GoodBehaviourVote = iota + GoodBehaviourBlockPart + ... +) +``` + +As a first iteration we provide a concrete implementation which wraps +the switch: +```go +type SwitchedPeerBehaviour struct { + sw *Switch +} + +func (spb *SwitchedPeerBehaviour) Errored(peer Peer, reason ErrorBehaviourPeer) { + spb.sw.StopPeerForError(peer, reason) +} + +func (spb *SwitchedPeerBehaviour) Behaved(peer Peer, reason GoodBehaviourPeer) { + spb.sw.MarkPeerAsGood(peer) +} + +func NewSwitchedPeerBehaviour(sw *Switch) *SwitchedPeerBehaviour { + return &SwitchedPeerBehaviour{ + sw: sw, + } +} +``` + +Reactors, which are often difficult to unit test[2](#references) could use an implementation which exposes the signals produced by the reactor in +manufactured scenarios: + +```go +type ErrorBehaviours map[Peer][]ErrorBehaviourPeer +type GoodBehaviours map[Peer][]GoodBehaviourPeer + +type StorePeerBehaviour struct { + eb ErrorBehaviours + gb GoodBehaviours +} + +func NewStorePeerBehaviour() *StorePeerBehaviour{ + return &StorePeerBehaviour{ + eb: make(ErrorBehaviours), + gb: make(GoodBehaviours), + } +} + +func (spb StorePeerBehaviour) Errored(peer Peer, reason ErrorBehaviourPeer) { + if _, ok := spb.eb[peer]; !ok { + spb.eb[peer] = []ErrorBehaviours{reason} + } else { + spb.eb[peer] = append(spb.eb[peer], reason) + } +} + +func (mpb *StorePeerBehaviour) GetErrored() ErrorBehaviours { + return mpb.eb +} + + +func (spb StorePeerBehaviour) Behaved(peer Peer, reason GoodBehaviourPeer) { + if _, ok := spb.gb[peer]; !ok { + spb.gb[peer] = []GoodBehaviourPeer{reason} + } else { + spb.gb[peer] = append(spb.gb[peer], reason) + } +} + +func (spb *StorePeerBehaviour) GetBehaved() GoodBehaviours { + return spb.gb +} +``` + +## Status + +Accepted + +## Consequences + +### Positive + + * De-couple signaling from acting upon peer behavior. + * Reduce the coupling of reactors and the Switch and the network + stack + * The responsibility of managing peer behavior can be migrated to + a single component instead of split between the switch and the + address book. + +### Negative + + * The first iteration will simply wrap the Switch and introduce a + level of indirection. + +### Neutral + +## References + +1. Issue [#2067](https://github.com/tendermint/tendermint/issues/2067): P2P Refactor +2. PR: [#3506](https://github.com/tendermint/tendermint/pull/3506): ADR 036: Blockchain Reactor Refactor diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-040-blockchain-reactor-refactor.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-040-blockchain-reactor-refactor.md new file mode 100644 index 00000000..520d55b5 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-040-blockchain-reactor-refactor.md @@ -0,0 +1,534 @@ +# ADR 040: Blockchain Reactor Refactor + +## Changelog + +19-03-2019: Initial draft + +## Context + +The Blockchain Reactor's high level responsibility is to enable peers who are far behind the current state of the +blockchain to quickly catch up by downloading many blocks in parallel from its peers, verifying block correctness, and +executing them against the ABCI application. We call the protocol executed by the Blockchain Reactor `fast-sync`. +The current architecture diagram of the blockchain reactor can be found here: + +![Blockchain Reactor Architecture Diagram](img/bc-reactor.png) + +The current architecture consists of dozens of routines and it is tightly depending on the `Switch`, making writing +unit tests almost impossible. Current tests require setting up complex dependency graphs and dealing with concurrency. +Note that having dozens of routines is in this case overkill as most of the time routines sits idle waiting for +something to happen (message to arrive or timeout to expire). Due to dependency on the `Switch`, testing relatively +complex network scenarios and failures (for example adding and removing peers) is very complex tasks and frequently lead +to complex tests with not deterministic behavior ([#3400]). Impossibility to write proper tests makes confidence in +the code low and this resulted in several issues (some are fixed in the meantime and some are still open): +[#3400], [#2897], [#2896], [#2699], [#2888], [#2457], [#2622], [#2026]. + +## Decision + +To remedy these issues we plan a major refactor of the blockchain reactor. The proposed architecture is largely inspired +by ADR-30 and is presented on the following diagram: +![Blockchain Reactor Refactor Diagram](img/bc-reactor-refactor.png) + +We suggest a concurrency architecture where the core algorithm (we call it `Controller`) is extracted into a finite +state machine. The active routine of the reactor is called `Executor` and is responsible for receiving and sending +messages from/to peers and triggering timeouts. What messages should be sent and timeouts triggered is determined mostly +by the `Controller`. The exception is `Peer Heartbeat` mechanism which is `Executor` responsibility. The heartbeat +mechanism is used to remove slow and unresponsive peers from the peer list. Writing of unit tests is simpler with +this architecture as most of the critical logic is part of the `Controller` function. We expect that simpler concurrency +architecture will not have significant negative effect on the performance of this reactor (to be confirmed by +experimental evaluation). + + +### Implementation changes + +We assume the following system model for "fast sync" protocol: + +* a node is connected to a random subset of all nodes that represents its peer set. Some nodes are correct and some + might be faulty. We don't make assumptions about ratio of faulty nodes, i.e., it is possible that all nodes in some + peer set are faulty. +* we assume that communication between correct nodes is synchronous, i.e., if a correct node `p` sends a message `m` to + a correct node `q` at time `t`, then `q` will receive message the latest at time `t+Delta` where `Delta` is a system + parameter that is known by network participants. `Delta` is normally chosen to be an order of magnitude higher than + the real communication delay (maximum) between correct nodes. Therefore if a correct node `p` sends a request message + to a correct node `q` at time `t` and there is no the corresponding reply at time `t + 2*Delta`, then `p` can assume + that `q` is faulty. Note that the network assumptions for the consensus reactor are different (we assume partially + synchronous model there). + +The requirements for the "fast sync" protocol are formally specified as follows: + +- `Correctness`: If a correct node `p` is connected to a correct node `q` for a long enough period of time, then `p` +- will eventually download all requested blocks from `q`. +- `Termination`: If a set of peers of a correct node `p` is stable (no new nodes are added to the peer set of `p`) for +- a long enough period of time, then protocol eventually terminates. +- `Fairness`: A correct node `p` sends requests for blocks to all peers from its peer set. + +As explained above, the `Executor` is responsible for sending and receiving messages that are part of the `fast-sync` +protocol. The following messages are exchanged as part of `fast-sync` protocol: + +``` go +type Message int +const ( + MessageUnknown Message = iota + MessageStatusRequest + MessageStatusResponse + MessageBlockRequest + MessageBlockResponse +) +``` +`MessageStatusRequest` is sent periodically to all peers as a request for a peer to provide its current height. It is +part of the `Peer Heartbeat` mechanism and a failure to respond timely to this message results in a peer being removed +from the peer set. Note that the `Peer Heartbeat` mechanism is used only while a peer is in `fast-sync` mode. We assume +here existence of a mechanism that gives node a possibility to inform its peers that it is in the `fast-sync` mode. + +``` go +type MessageStatusRequest struct { + SeqNum int64 // sequence number of the request +} +``` +`MessageStatusResponse` is sent as a response to `MessageStatusRequest` to inform requester about the peer current +height. + +``` go +type MessageStatusResponse struct { + SeqNum int64 // sequence number of the corresponding request + Height int64 // current peer height +} +``` + +`MessageBlockRequest` is used to make a request for a block and the corresponding commit certificate at a given height. + +``` go +type MessageBlockRequest struct { + Height int64 +} +``` + +`MessageBlockResponse` is a response for the corresponding block request. In addition to providing the block and the +corresponding commit certificate, it contains also a current peer height. + +``` go +type MessageBlockResponse struct { + Height int64 + Block Block + Commit Commit + PeerHeight int64 +} +``` + +In addition to sending and receiving messages, and `HeartBeat` mechanism, controller is also managing timeouts +that are triggered upon `Controller` request. `Controller` is then informed once a timeout expires. + +``` go +type TimeoutTrigger int +const ( + TimeoutUnknown TimeoutTrigger = iota + TimeoutResponseTrigger + TimeoutTerminationTrigger +) +``` + +The `Controller` can be modelled as a function with clearly defined inputs: + +* `State` - current state of the node. Contains data about connected peers and its behavior, pending requests, +* received blocks, etc. +* `Event` - significant events in the network. + +producing clear outputs: + +* `State` - updated state of the node, +* `MessageToSend` - signal what message to send and to which peer +* `TimeoutTrigger` - signal that timeout should be triggered. + + +We consider the following `Event` types: + +``` go +type Event int +const ( + EventUnknown Event = iota + EventStatusReport + EventBlockRequest + EventBlockResponse + EventRemovePeer + EventTimeoutResponse + EventTimeoutTermination +) +``` + +`EventStatusResponse` event is generated once `MessageStatusResponse` is received by the `Executor`. + +``` go +type EventStatusReport struct { + PeerID ID + Height int64 +} +``` + +`EventBlockRequest` event is generated once `MessageBlockRequest` is received by the `Executor`. + +``` go +type EventBlockRequest struct { + Height int64 + PeerID p2p.ID +} +``` +`EventBlockResponse` event is generated upon reception of `MessageBlockResponse` message by the `Executor`. + +``` go +type EventBlockResponse struct { + Height int64 + Block Block + Commit Commit + PeerID ID + PeerHeight int64 +} +``` +`EventRemovePeer` is generated by `Executor` to signal that the connection to a peer is closed due to peer misbehavior. + +``` go +type EventRemovePeer struct { + PeerID ID +} +``` +`EventTimeoutResponse` is generated by `Executor` to signal that a timeout triggered by `TimeoutResponseTrigger` has +expired. + +``` go +type EventTimeoutResponse struct { + PeerID ID + Height int64 +} +``` +`EventTimeoutTermination` is generated by `Executor` to signal that a timeout triggered by `TimeoutTerminationTrigger` +has expired. + +``` go +type EventTimeoutTermination struct { + Height int64 +} +``` + +`MessageToSend` is just a wrapper around `Message` type that contains id of the peer to which message should be sent. + +``` go +type MessageToSend struct { + PeerID ID + Message Message +} +``` + +The Controller state machine can be in two modes: `ModeFastSync` when +a node is trying to catch up with the network by downloading committed blocks, +and `ModeConsensus` in which it executes Tendermint consensus protocol. We +consider that `fast sync` mode terminates once the Controller switch to +`ModeConsensus`. + +``` go +type Mode int +const ( + ModeUnknown Mode = iota + ModeFastSync + ModeConsensus +) +``` +`Controller` is managing the following state: + +``` go +type ControllerState struct { + Height int64 // the first block that is not committed + Mode Mode // mode of operation + PeerMap map[ID]PeerStats // map of peer IDs to peer statistics + MaxRequestPending int64 // maximum height of the pending requests + FailedRequests []int64 // list of failed block requests + PendingRequestsNum int // total number of pending requests + Store []BlockInfo // contains list of downloaded blocks + Executor BlockExecutor // store, verify and executes blocks +} +``` + +`PeerStats` data structure keeps for every peer its current height and a list of pending requests for blocks. + +``` go +type PeerStats struct { + Height int64 + PendingRequest int64 // a request sent to this peer +} +``` + +`BlockInfo` data structure is used to store information (as part of block store) about downloaded blocks: from what peer + a block and the corresponding commit certificate are received. +``` go +type BlockInfo struct { + Block Block + Commit Commit + PeerID ID // a peer from which we received the corresponding Block and Commit +} +``` + +The `Controller` is initialized by providing an initial height (`startHeight`) from which it will start downloading +blocks from peers and the current state of the `BlockExecutor`. + +``` go +func NewControllerState(startHeight int64, executor BlockExecutor) ControllerState { + state = ControllerState {} + state.Height = startHeight + state.Mode = ModeFastSync + state.MaxRequestPending = startHeight - 1 + state.PendingRequestsNum = 0 + state.Executor = executor + initialize state.PeerMap, state.FailedRequests and state.Store to empty data structures + return state +} +``` + +The core protocol logic is given with the following function: + +``` go +func handleEvent(state ControllerState, event Event) (ControllerState, Message, TimeoutTrigger, Error) { + msg = nil + timeout = nil + error = nil + + switch state.Mode { + case ModeConsensus: + switch event := event.(type) { + case EventBlockRequest: + msg = createBlockResponseMessage(state, event) + return state, msg, timeout, error + default: + error = "Only respond to BlockRequests while in ModeConsensus!" + return state, msg, timeout, error + } + + case ModeFastSync: + switch event := event.(type) { + case EventBlockRequest: + msg = createBlockResponseMessage(state, event) + return state, msg, timeout, error + + case EventStatusResponse: + return handleEventStatusResponse(event, state) + + case EventRemovePeer: + return handleEventRemovePeer(event, state) + + case EventBlockResponse: + return handleEventBlockResponse(event, state) + + case EventResponseTimeout: + return handleEventResponseTimeout(event, state) + + case EventTerminationTimeout: + // Termination timeout is triggered in case of empty peer set and in case there are no pending requests. + // If this timeout expires and in the meantime no new peers are added or new pending requests are made + // then `fast-sync` mode terminates by switching to `ModeConsensus`. + // Note that termination timeout should be higher than the response timeout. + if state.Height == event.Height && state.PendingRequestsNum == 0 { state.State = ConsensusMode } + return state, msg, timeout, error + + default: + error = "Received unknown event type!" + return state, msg, timeout, error + } + } +} +``` + +``` go +func createBlockResponseMessage(state ControllerState, event BlockRequest) MessageToSend { + msgToSend = nil + if _, ok := state.PeerMap[event.PeerID]; !ok { peerStats = PeerStats{-1, -1} } + if state.Executor.ContainsBlockWithHeight(event.Height) && event.Height > peerStats.Height { + peerStats = event.Height + msg = BlockResponseMessage{ + Height: event.Height, + Block: state.Executor.getBlock(eventHeight), + Commit: state.Executor.getCommit(eventHeight), + PeerID: event.PeerID, + CurrentHeight: state.Height - 1, + } + msgToSend = MessageToSend { event.PeerID, msg } + } + state.PeerMap[event.PeerID] = peerStats + return msgToSend +} +``` + +``` go +func handleEventStatusResponse(event EventStatusResponse, state ControllerState) (ControllerState, MessageToSend, TimeoutTrigger, Error) { + if _, ok := state.PeerMap[event.PeerID]; !ok { + peerStats = PeerStats{ -1, -1 } + } else { + peerStats = state.PeerMap[event.PeerID] + } + + if event.Height > peerStats.Height { peerStats.Height = event.Height } + // if there are no pending requests for this peer, try to send him a request for block + if peerStats.PendingRequest == -1 { + msg = createBlockRequestMessages(state, event.PeerID, peerStats.Height) + // msg is nil if no request for block can be made to a peer at this point in time + if msg != nil { + peerStats.PendingRequests = msg.Height + state.PendingRequestsNum++ + // when a request for a block is sent to a peer, a response timeout is triggered. If no corresponding block is sent by the peer + // during response timeout period, then the peer is considered faulty and is removed from the peer set. + timeout = ResponseTimeoutTrigger{ msg.PeerID, msg.Height, PeerTimeout } + } else if state.PendingRequestsNum == 0 { + // if there are no pending requests and no new request can be placed to the peer, termination timeout is triggered. + // If termination timeout expires and we are still at the same height and there are no pending requests, the "fast-sync" + // mode is finished and we switch to `ModeConsensus`. + timeout = TerminationTimeoutTrigger{ state.Height, TerminationTimeout } + } + } + state.PeerMap[event.PeerID] = peerStats + return state, msg, timeout, error +} +``` + +``` go +func handleEventRemovePeer(event EventRemovePeer, state ControllerState) (ControllerState, MessageToSend, TimeoutTrigger, Error) { + if _, ok := state.PeerMap[event.PeerID]; ok { + pendingRequest = state.PeerMap[event.PeerID].PendingRequest + // if a peer is removed from the peer set, its pending request is declared failed and added to the `FailedRequests` list + // so it can be retried. + if pendingRequest != -1 { + add(state.FailedRequests, pendingRequest) + } + state.PendingRequestsNum-- + delete(state.PeerMap, event.PeerID) + // if the peer set is empty after removal of this peer then termination timeout is triggered. + if state.PeerMap.isEmpty() { + timeout = TerminationTimeoutTrigger{ state.Height, TerminationTimeout } + } + } else { error = "Removing unknown peer!" } + return state, msg, timeout, error +``` + +``` go +func handleEventBlockResponse(event EventBlockResponse, state ControllerState) (ControllerState, MessageToSend, TimeoutTrigger, Error) + if state.PeerMap[event.PeerID] { + peerStats = state.PeerMap[event.PeerID] + // when expected block arrives from a peer, it is added to the store so it can be verified and if correct executed after. + if peerStats.PendingRequest == event.Height { + peerStats.PendingRequest = -1 + state.PendingRequestsNum-- + if event.PeerHeight > peerStats.Height { peerStats.Height = event.PeerHeight } + state.Store[event.Height] = BlockInfo{ event.Block, event.Commit, event.PeerID } + // blocks are verified sequentially so adding a block to the store does not mean that it will be immediately verified + // as some of the previous blocks might be missing. + state = verifyBlocks(state) // it can lead to event.PeerID being removed from peer list + if _, ok := state.PeerMap[event.PeerID]; ok { + // we try to identify new request for a block that can be asked to the peer + msg = createBlockRequestMessage(state, event.PeerID, peerStats.Height) + if msg != nil { + peerStats.PendingRequests = msg.Height + state.PendingRequestsNum++ + // if request for block is made, response timeout is triggered + timeout = ResponseTimeoutTrigger{ msg.PeerID, msg.Height, PeerTimeout } + } else if state.PeerMap.isEmpty() || state.PendingRequestsNum == 0 { + // if the peer map is empty (the peer can be removed as block verification failed) or there are no pending requests + // termination timeout is triggered. + timeout = TerminationTimeoutTrigger{ state.Height, TerminationTimeout } + } + } + } else { error = "Received Block from wrong peer!" } + } else { error = "Received Block from unknown peer!" } + + state.PeerMap[event.PeerID] = peerStats + return state, msg, timeout, error +} +``` + +``` go +func handleEventResponseTimeout(event, state) { + if _, ok := state.PeerMap[event.PeerID]; ok { + peerStats = state.PeerMap[event.PeerID] + // if a response timeout expires and the peer hasn't delivered the block, the peer is removed from the peer list and + // the request is added to the `FailedRequests` so the block can be downloaded from other peer + if peerStats.PendingRequest == event.Height { + add(state.FailedRequests, pendingRequest) + delete(state.PeerMap, event.PeerID) + state.PendingRequestsNum-- + // if peer set is empty, then termination timeout is triggered + if state.PeerMap.isEmpty() { + timeout = TimeoutTrigger{ state.Height, TerminationTimeout } + } + } + } + return state, msg, timeout, error +} +``` + +``` go +func createBlockRequestMessage(state ControllerState, peerID ID, peerHeight int64) MessageToSend { + msg = nil + blockHeight = -1 + r = find request in state.FailedRequests such that r <= peerHeight // returns `nil` if there are no such request + // if there is a height in failed requests that can be downloaded from the peer send request to it + if r != nil { + blockNumber = r + delete(state.FailedRequests, r) + } else if state.MaxRequestPending < peerHeight { + // if height of the maximum pending request is smaller than peer height, then ask peer for next block + state.MaxRequestPending++ + blockHeight = state.MaxRequestPending // increment state.MaxRequestPending and then return the new value + } + + if blockHeight > -1 { msg = MessageToSend { peerID, MessageBlockRequest { blockHeight } } + return msg +} +``` + +``` go +func verifyBlocks(state State) State { + done = false + for !done { + block = state.Store[height] + if block != nil { + verified = verify block.Block using block.Commit // return `true` is verification succeed, 'false` otherwise + + if verified { + block.Execute() // executing block is costly operation so it might make sense executing asynchronously + state.Height++ + } else { + // if block verification failed, then it is added to `FailedRequests` and the peer is removed from the peer set + add(state.FailedRequests, height) + state.Store[height] = nil + if _, ok := state.PeerMap[block.PeerID]; ok { + pendingRequest = state.PeerMap[block.PeerID].PendingRequest + // if there is a pending request sent to the peer that is just to be removed from the peer set, add it to `FailedRequests` + if pendingRequest != -1 { + add(state.FailedRequests, pendingRequest) + state.PendingRequestsNum-- + } + delete(state.PeerMap, event.PeerID) + } + done = true + } + } else { done = true } + } + return state +} +``` + +In the proposed architecture `Controller` is not active task, i.e., it is being called by the `Executor`. Depending on +the return values returned by `Controller`,`Executor` will send a message to some peer (`msg` != nil), trigger a +timeout (`timeout` != nil) or deal with errors (`error` != nil). +In case a timeout is triggered, it will provide as an input to `Controller` the corresponding timeout event once +timeout expires. + + +## Status + +Draft. + +## Consequences + +### Positive + +- isolated implementation of the algorithm +- improved testability - simpler to prove correctness +- clearer separation of concerns - easier to reason + +### Negative + +### Neutral diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-041-proposer-selection-via-abci.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-041-proposer-selection-via-abci.md new file mode 100644 index 00000000..58bf20de --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-041-proposer-selection-via-abci.md @@ -0,0 +1,29 @@ +# ADR 041: Application should be in charge of validator set + +## Changelog + + +## Context + +Currently Tendermint is in charge of validator set and proposer selection. Application can only update the validator set changes at EndBlock time. +To support Light Client, application should make sure at least 2/3 of validator are same at each round. + +Application should have full control on validator set changes and proposer selection. In each round Application can provide the list of validators for next rounds in order with their power. The proposer is the first in the list, in case the proposer is offline, the next one can propose the proposal and so on. + +## Decision + +## Status + +## Consequences + +Tendermint is no more in charge of validator set and its changes. The Application should provide the correct information. +However Tendermint can provide psedo-randomness algorithm to help application for selecting proposer in each round. + +### Positive + +### Negative + +### Neutral + +## References + diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-042-state-sync.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-042-state-sync.md new file mode 100644 index 00000000..a1589318 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-042-state-sync.md @@ -0,0 +1,235 @@ +# ADR 042: State Sync Design + +## Changelog + +2019-06-27: Init by EB +2019-07-04: Follow up by brapse + +## Context +StateSync is a feature which would allow a new node to receive a +snapshot of the application state without downloading blocks or going +through consensus. Once downloaded, the node could switch to FastSync +and eventually participate in consensus. The goal of StateSync is to +facilitate setting up a new node as quickly as possible. + +## Considerations +Because Tendermint doesn't know anything about the application state, +StateSync will broker messages between nodes and through +the ABCI to an opaque applicaton. The implementation will have multiple +touch points on both the tendermint code base and ABCI application. + +* A StateSync reactor to facilitate peer communication - Tendermint +* A Set of ABCI messages to transmit application state to the reactor - Tendermint +* A Set of MultiStore APIs for exposing snapshot data to the ABCI - ABCI application +* A Storage format with validation and performance considerations - ABCI application + +### Implementation Properties +Beyond the approach, any implementation of StateSync can be evaluated +across different criteria: + +* Speed: Expected throughput of producing and consuming snapshots +* Safety: Cost of pushing invalid snapshots to a node +* Liveness: Cost of preventing a node from receiving/constructing a snapshot +* Effort: How much effort does an implementation require + +### Implementation Question +* What is the format of a snapshot + * Complete snapshot + * Ordered IAVL key ranges + * Compressed individually chunks which can be validated +* How is data validated + * Trust a peer with it's data blindly + * Trust a majority of peers + * Use light client validation to validate each chunk against consensus + produced merkle tree root +* What are the performance characteristics + * Random vs sequential reads + * How parallelizeable is the scheduling algorithm + +### Proposals +Broadly speaking there are two approaches to this problem which have had +varying degrees of discussion and progress. These approach can be +summarized as: + +**Lazy:** Where snapshots are produced dynamically at request time. This +solution would use the existing data structure. +**Eager:** Where snapshots are produced periodically and served from disk at +request time. This solution would create an auxiliary data structure +optimized for batch read/writes. + +Additionally the propsosals tend to vary on how they provide safety +properties. + +**LightClient** Where a client can aquire the merkle root from the block +headers synchronized from a trusted validator set. Subsets of the application state, +called chunks can therefore be validated on receipt to ensure each chunk +is part of the merkle root. + +**Majority of Peers** Where manifests of chunks along with checksums are +downloaded and compared against versions provided by a majority of +peers. + +#### Lazy StateSync +An initial specification was published by Alexis Sellier. +In this design, the state has a given `size` of primitive elements (like +keys or nodes), each element is assigned a number from 0 to `size-1`, +and chunks consists of a range of such elements. Ackratos raised +[some concerns](https://docs.google.com/document/d/1npGTAa1qxe8EQZ1wG0a0Sip9t5oX2vYZNUDwr_LVRR4/edit) +about this design, somewhat specific to the IAVL tree, and mainly concerning +performance of random reads and of iterating through the tree to determine element numbers +(ie. elements aren't indexed by the element number). + +An alternative design was suggested by Jae Kwon in +[#3639](https://github.com/tendermint/tendermint/issues/3639) where chunking +happens lazily and in a dynamic way: nodes request key ranges from their peers, +and peers respond with some subset of the +requested range and with notes on how to request the rest in parallel from other +peers. Unlike chunk numbers, keys can be verified directly. And if some keys in the +range are ommitted, proofs for the range will fail to verify. +This way a node can start by requesting the entire tree from one peer, +and that peer can respond with say the first few keys, and the ranges to request +from other peers. + +Additionally, per chunk validation tends to come more naturally to the +Lazy approach since it tends to use the existing structure of the tree +(ie. keys or nodes) rather than state-sync specific chunks. Such a +design for tendermint was originally tracked in +[#828](https://github.com/tendermint/tendermint/issues/828). + +#### Eager StateSync +Warp Sync as implemented in OpenEthereum to rapidly +download both blocks and state snapshots from peers. Data is carved into ~4MB +chunks and snappy compressed. Hashes of snappy compressed chunks are stored in a +manifest file which co-ordinates the state-sync. Obtaining a correct manifest +file seems to require an honest majority of peers. This means you may not find +out the state is incorrect until you download the whole thing and compare it +with a verified block header. + +A similar solution was implemented by Binance in +[#3594](https://github.com/tendermint/tendermint/pull/3594) +based on their initial implementation in +[PR #3243](https://github.com/tendermint/tendermint/pull/3243) +and [some learnings](https://docs.google.com/document/d/1npGTAa1qxe8EQZ1wG0a0Sip9t5oX2vYZNUDwr_LVRR4/edit). +Note this still requires the honest majority peer assumption. + +As an eager protocol, warp-sync can efficiently compress larger, more +predicatable chunks once per snapshot and service many new peers. By +comparison lazy chunkers would have to compress each chunk at request +time. + +### Analysis of Lazy vs Eager +Lazy vs Eager have more in common than they differ. They all require +reactors on the tendermint side, a set of ABCI messages and a method for +serializing/deserializing snapshots facilitated by a SnapshotFormat. + +The biggest difference between Lazy and Eager proposals is in the +read/write patterns necessitated by serving a snapshot chunk. +Specifically, Lazy State Sync performs random reads to the underlying data +structure while Eager can optimize for sequential reads. + +This distinctin between approaches was demonstrated by Binance's +[ackratos](https://github.com/ackratos) in their implementation of [Lazy +State sync](https://github.com/tendermint/tendermint/pull/3243), The +[analysis](https://docs.google.com/document/d/1npGTAa1qxe8EQZ1wG0a0Sip9t5oX2vYZNUDwr_LVRR4/) +of the performance, and follow up implementation of [Warp +Sync](http://github.com/tendermint/tendermint/pull/3594). + +#### Compairing Security Models +There are several different security models which have been +discussed/proposed in the past but generally fall into two categories. + +Light client validation: In which the node receiving data is expected to +first perform a light client sync and have all the nessesary block +headers. Within the trusted block header (trusted in terms of from a +validator set subject to [weak +subjectivity](https://github.com/tendermint/tendermint/pull/3795)) and +can compare any subset of keys called a chunk against the merkle root. +The advantage of light client validation is that the block headers are +signed by validators which have something to lose for malicious +behavior. If a validator were to provide an invalid proof, they can be +slashed. + +Majority of peer validation: A manifest file containing a list of chunks +along with checksums of each chunk is downloaded from a +trusted source. That source can be a community resource similar to +[sum.golang.org](https://sum.golang.org) or downloaded from the majority +of peers. One disadantage of the majority of peer security model is the +vuliberability to eclipse attacks in which a malicious users looks to +saturate a target node's peer list and produce a manufactured picture of +majority. + +A third option would be to include snapshot related data in the +block header. This could include the manifest with related checksums and be +secured through consensus. One challenge of this approach is to +ensure that creating snapshots does not put undo burden on block +propsers by synchronizing snapshot creation and block creation. One +approach to minimizing the burden is for snapshots for height +`H` to be included in block `H+n` where `n` is some `n` block away, +giving the block propser enough time to complete the snapshot +asynchronousy. + +## Proposal: Eager StateSync With Per Chunk Light Client Validation +The conclusion after some concideration of the advantages/disadvances of +eager/lazy and different security models is to produce a state sync +which eagerly produces snapshots and uses light client validation. This +approach has the performance advantages of pre-computing efficient +snapshots which can streamed to new nodes on demand using sequential IO. +Secondly, by using light client validation we cna validate each chunk on +receipt and avoid the potential eclipse attack of majority of peer based +security. + +### Implementation +Tendermint is responsible for downloading and verifying chunks of +AppState from peers. ABCI Application is responsible for taking +AppStateChunk objects from TM and constructing a valid state tree whose +root corresponds with the AppHash of syncing block. In particular we +will need implement: + +* Build new StateSync reactor brokers message transmission between the peers + and the ABCI application +* A set of ABCI Messages +* Design SnapshotFormat as an interface which can: + * validate chunks + * read/write chunks from file + * read/write chunks to/from application state store + * convert manifests into chunkRequest ABCI messages +* Implement SnapshotFormat for cosmos-hub with concrete implementation for: + * read/write chunks in a way which can be: + * parallelized across peers + * validated on receipt + * read/write to/from IAVL+ tree + +![StateSync Architecture Diagram](img/state-sync.png) + +## Implementation Path +* Create StateSync reactor based on [#3753](https://github.com/tendermint/tendermint/pull/3753) +* Design SnapshotFormat with an eye towards cosmos-hub implementation +* ABCI message to send/receive SnapshotFormat +* IAVL+ changes to support SnapshotFormat +* Deliver Warp sync (no chunk validation) +* light client implementation for weak subjectivity +* Deliver StateSync with chunk validation + +## Status + +Proposed + +## Concequences + +### Neutral + +### Positive +* Safe & performant state sync design substantiated with real world implementation experience +* General interfaces allowing application specific innovation +* Parallizable implementation trajectory with reasonable engineering effort + +### Negative +* Static Scheduling lacks opportunity for real time chunk availability optimizations + +## References +[sync: Sync current state without full replay for Applications](https://github.com/tendermint/tendermint/issues/828) - original issue +[tendermint state sync proposal 2](https://docs.google.com/document/d/1npGTAa1qxe8EQZ1wG0a0Sip9t5oX2vYZNUDwr_LVRR4/edit) - ackratos proposal +[proposal 2 implementation](https://github.com/tendermint/tendermint/pull/3243) - ackratos implementation +[WIP General/Lazy State-Sync pseudo-spec](https://github.com/tendermint/tendermint/issues/3639) - Jae Proposal +[Warp Sync Implementation](https://github.com/tendermint/tendermint/pull/3594) - ackratos +[Chunk Proposal](https://github.com/tendermint/tendermint/pull/3799) - Bucky proposed diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-043-blockchain-riri-org.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-043-blockchain-riri-org.md new file mode 100644 index 00000000..dbe04eea --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-043-blockchain-riri-org.md @@ -0,0 +1,404 @@ +# ADR 043: Blockhchain Reactor Riri-Org + +## Changelog + +- 18-06-2019: Initial draft +- 08-07-2019: Reviewed +- 29-11-2019: Implemented +- 14-02-2020: Updated with the implementation details + +## Context + +The blockchain reactor is responsible for two high level processes:sending/receiving blocks from peers and FastSync-ing blocks to catch upnode who is far behind. The goal of [ADR-40](https://github.com/tendermint/tendermint/blob/main/docs/architecture/adr-040-blockchain-reactor-refactor.md) was to refactor these two processes by separating business logic currently wrapped up in go-channels into pure `handle*` functions. While the ADR specified what the final form of the reactor might look like it lacked guidance on intermediary steps to get there. +The following diagram illustrates the state of the [blockchain-reorg](https://github.com/tendermint/tendermint/pull/3561) reactor which will be referred to as `v1`. + +![v1 Blockchain Reactor Architecture +Diagram](https://github.com/tendermint/tendermint/blob/f9e556481654a24aeb689bdadaf5eab3ccd66829/docs/architecture/img/blockchain-reactor-v1.png) + +While `v1` of the blockchain reactor has shown significant improvements in terms of simplifying the concurrency model, the current PR has run into few roadblocks. + +- The current PR large and difficult to review. +- Block gossiping and fast sync processes are highly coupled to the shared `Pool` data structure. +- Peer communication is spread over multiple components creating complex dependency graph which must be mocked out during testing. +- Timeouts modeled as stateful tickers introduce non-determinism in tests + +This ADR is meant to specify the missing components and control necessary to achieve [ADR-40](https://github.com/tendermint/tendermint/blob/main/docs/architecture/adr-040-blockchain-reactor-refactor.md). + +## Decision + +Partition the responsibilities of the blockchain reactor into a set of components which communicate exclusively with events. Events will contain timestamps allowing each component to track time as internal state. The internal state will be mutated by a set of `handle*` which will produce event(s). The integration between components will happen in the reactor and reactor tests will then become integration tests between components. This design will be known as `v2`. + +![v2 Blockchain Reactor Architecture +Diagram](https://github.com/tendermint/tendermint/blob/584e67ac3fac220c5c3e0652e3582eca8231e814/docs/architecture/img/blockchain-reactor-v2.png) + +### Fast Sync Related Communication Channels + +The diagram below shows the fast sync routines and the types of channels and queues used to communicate with each other. +In addition the per reactor channels used by the sendRoutine to send messages over the Peer MConnection are shown. + +![v2 Blockchain Channels and Queues +Diagram](https://github.com/tendermint/tendermint/blob/5cf570690f989646fb3b615b734da503f038891f/docs/architecture/img/blockchain-v2-channels.png) + +### Reactor changes in detail + +The reactor will include a demultiplexing routine which will send each message to each sub routine for independent processing. Each sub routine will then select the messages it's interested in and call the handle specific function specified in [ADR-40](https://github.com/tendermint/tendermint/blob/main/docs/architecture/adr-040-blockchain-reactor-refactor.md). The demuxRoutine acts as "pacemaker" setting the time in which events are expected to be handled. + +```go +func demuxRoutine(msgs, scheduleMsgs, processorMsgs, ioMsgs) { + timer := time.NewTicker(interval) + for { + select { + case <-timer.C: + now := evTimeCheck{time.Now()} + schedulerMsgs <- now + processorMsgs <- now + ioMsgs <- now + case msg:= <- msgs: + msg.time = time.Now() + // These channels should produce backpressure before + // being full to avoid starving each other + schedulerMsgs <- msg + processorMsgs <- msg + ioMesgs <- msg + if msg == stop { + break; + } + } + } +} + +func processRoutine(input chan Message, output chan Message) { + processor := NewProcessor(..) + for { + msg := <- input + switch msg := msg.(type) { + case bcBlockRequestMessage: + output <- processor.handleBlockRequest(msg)) + ... + case stop: + processor.stop() + break; + } +} + +func scheduleRoutine(input chan Message, output chan Message) { + schelduer = NewScheduler(...) + for { + msg := <-msgs + switch msg := input.(type) { + case bcBlockResponseMessage: + output <- scheduler.handleBlockResponse(msg) + ... + case stop: + schedule.stop() + break; + } + } +} +``` + +## Lifecycle management + +A set of routines for individual processes allow processes to run in parallel with clear lifecycle management. `Start`, `Stop`, and `AddPeer` hooks currently present in the reactor will delegate to the sub-routines allowing them to manage internal state independent without further coupling to the reactor. + +```go +func (r *BlockChainReactor) Start() { + r.msgs := make(chan Message, maxInFlight) + schedulerMsgs := make(chan Message) + processorMsgs := make(chan Message) + ioMsgs := make(chan Message) + + go processorRoutine(processorMsgs, r.msgs) + go scheduleRoutine(schedulerMsgs, r.msgs) + go ioRoutine(ioMsgs, r.msgs) + ... +} + +func (bcR *BlockchainReactor) Receive(...) { + ... + r.msgs <- msg + ... +} + +func (r *BlockchainReactor) Stop() { + ... + r.msgs <- stop + ... +} + +... +func (r *BlockchainReactor) Stop() { + ... + r.msgs <- stop + ... +} +... + +func (r *BlockchainReactor) AddPeer(peer p2p.Peer) { + ... + r.msgs <- bcAddPeerEv{peer.ID} + ... +} + +``` + +## IO handling + +An io handling routine within the reactor will isolate peer communication. Message going through the ioRoutine will usually be one way, using `p2p` APIs. In the case in which the `p2p` API such as `trySend` return errors, the ioRoutine can funnel those message back to the demuxRoutine for distribution to the other routines. For instance errors from the ioRoutine can be consumed by the scheduler to inform better peer selection implementations. + +```go +func (r *BlockchainReacor) ioRoutine(ioMesgs chan Message, outMsgs chan Message) { + ... + for { + msg := <-ioMsgs + switch msg := msg.(type) { + case scBlockRequestMessage: + queued := r.sendBlockRequestToPeer(...) + if queued { + outMsgs <- ioSendQueued{...} + } + case scStatusRequestMessage + r.sendStatusRequestToPeer(...) + case bcPeerError + r.Swtich.StopPeerForError(msg.src) + ... + ... + case bcFinished + break; + } + } +} + +``` + +### Processor Internals + +The processor is responsible for ordering, verifying and executing blocks. The Processor will maintain an internal cursor `height` refering to the last processed block. As a set of blocks arrive unordered, the Processor will check if it has `height+1` necessary to process the next block. The processor also maintains the map `blockPeers` of peers to height, to keep track of which peer provided the block at `height`. `blockPeers` can be used in`handleRemovePeer(...)` to reschedule all unprocessed blocks provided by a peer who has errored. + +```go +type Processor struct { + height int64 // the height cursor + state ... + blocks [height]*Block // keep a set of blocks in memory until they are processed + blockPeers [height]PeerID // keep track of which heights came from which peerID + lastTouch timestamp +} + +func (proc *Processor) handleBlockResponse(peerID, block) { + if block.height <= height || block[block.height] { + } else if blocks[block.height] { + return errDuplicateBlock{} + } else { + blocks[block.height] = block + } + + if blocks[height] && blocks[height+1] { + ... = state.Validators.VerifyCommit(...) + ... = store.SaveBlock(...) + state, err = blockExec.ApplyBlock(...) + ... + if err == nil { + delete blocks[height] + height++ + lastTouch = msg.time + return pcBlockProcessed{height-1} + } else { + ... // Delete all unprocessed block from the peer + return pcBlockProcessError{peerID, height} + } + } +} + +func (proc *Processor) handleRemovePeer(peerID) { + events = [] + // Delete all unprocessed blocks from peerID + for i = height; i < len(blocks); i++ { + if blockPeers[i] == peerID { + events = append(events, pcBlockReschedule{height}) + + delete block[height] + } + } + return events +} + +func handleTimeCheckEv(time) { + if time - lastTouch > timeout { + // Timeout the processor + ... + } +} +``` + +## Schedule + +The Schedule maintains the internal state used for scheduling blockRequestMessages based on some scheduling algorithm. The schedule needs to maintain state on: + +- The state `blockState` of every block seem up to height of maxHeight +- The set of peers and their peer state `peerState` +- which peers have which blocks +- which blocks have been requested from which peers + +```go +type blockState int + +const ( + blockStateNew = iota + blockStatePending, + blockStateReceived, + blockStateProcessed +) + +type schedule { + // a list of blocks in which blockState + blockStates map[height]blockState + + // a map of which blocks are available from which peers + blockPeers map[height]map[p2p.ID]scPeer + + // a map of peerID to schedule specific peer struct `scPeer` + peers map[p2p.ID]scPeer + + // a map of heights to the peer we are waiting for a response from + pending map[height]scPeer + + targetPending int // the number of blocks we want in blockStatePending + targetReceived int // the number of blocks we want in blockStateReceived + + peerTimeout int + peerMinSpeed int +} + +func (sc *schedule) numBlockInState(state blockState) uint32 { + num := 0 + for i := sc.minHeight(); i <= sc.maxHeight(); i++ { + if sc.blockState[i] == state { + num++ + } + } + return num +} + + +func (sc *schedule) popSchedule(maxRequest int) []scBlockRequestMessage { + // We only want to schedule requests such that we have less than sc.targetPending and sc.targetReceived + // This ensures we don't saturate the network or flood the processor with unprocessed blocks + todo := min(sc.targetPending - sc.numBlockInState(blockStatePending), sc.numBlockInState(blockStateReceived)) + events := []scBlockRequestMessage{} + for i := sc.minHeight(); i < sc.maxMaxHeight(); i++ { + if todo == 0 { + break + } + if blockStates[i] == blockStateNew { + peer = sc.selectPeer(blockPeers[i]) + sc.blockStates[i] = blockStatePending + sc.pending[i] = peer + events = append(events, scBlockRequestMessage{peerID: peer.peerID, height: i}) + todo-- + } + } + return events +} +... + +type scPeer struct { + peerID p2p.ID + numOustandingRequest int + lastTouched time.Time + monitor flow.Monitor +} + +``` + +# Scheduler + +The scheduler is configured to maintain a target `n` of in flight +messages and will use feedback from `_blockResponseMessage`, +`_statusResponseMessage` and `_peerError` produce an optimal assignment +of scBlockRequestMessage at each `timeCheckEv`. + +``` + +func handleStatusResponse(peerID, height, time) { + schedule.touchPeer(peerID, time) + schedule.setPeerHeight(peerID, height) +} + +func handleBlockResponseMessage(peerID, height, block, time) { + schedule.touchPeer(peerID, time) + schedule.markReceived(peerID, height, size(block)) +} + +func handleNoBlockResponseMessage(peerID, height, time) { + schedule.touchPeer(peerID, time) + // reschedule that block, punish peer... + ... +} + +func handlePeerError(peerID) { + // Remove the peer, reschedule the requests + ... +} + +func handleTimeCheckEv(time) { + // clean peer list + + events = [] + for peerID := range schedule.peersNotTouchedSince(time) { + pending = schedule.pendingFrom(peerID) + schedule.setPeerState(peerID, timedout) + schedule.resetBlocks(pending) + events = append(events, peerTimeout{peerID}) + } + + events = append(events, schedule.popSchedule()) + + return events +} +``` + +## Peer + +The Peer Stores per peer state based on messages received by the scheduler. + +```go +type Peer struct { + lastTouched timestamp + lastDownloaded timestamp + pending map[height]struct{} + height height // max height for the peer + state { + pending, // we know the peer but not the height + active, // we know the height + timeout // the peer has timed out + } +} +``` + +## Status + +Implemented + +## Consequences + +### Positive + +- Test become deterministic +- Simulation becomes a-termporal: no need wait for a wall-time timeout +- Peer Selection can be independently tested/simulated +- Develop a general approach to refactoring reactors + +### Negative + +### Neutral + +### Implementation Path + +- Implement the scheduler, test the scheduler, review the rescheduler +- Implement the processor, test the processor, review the processor +- Implement the demuxer, write integration test, review integration tests + +## References + +- [ADR-40](https://github.com/tendermint/tendermint/blob/main/docs/architecture/adr-040-blockchain-reactor-refactor.md): The original blockchain reactor re-org proposal +- [Blockchain re-org](https://github.com/tendermint/tendermint/pull/3561): The current blockchain reactor re-org implementation (v1) diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-044-lite-client-with-weak-subjectivity.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-044-lite-client-with-weak-subjectivity.md new file mode 100644 index 00000000..9fbd808f --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-044-lite-client-with-weak-subjectivity.md @@ -0,0 +1,141 @@ +# ADR 044: Lite Client with Weak Subjectivity + +## Changelog +* 13-07-2019: Initial draft +* 14-08-2019: Address cwgoes comments + +## Context + +The concept of light clients was introduced in the Bitcoin white paper. It +describes a watcher of distributed consensus process that only validates the +consensus algorithm and not the state machine transactions within. + +Tendermint light clients allow bandwidth & compute-constrained devices, such as smartphones, low-power embedded chips, or other blockchains to +efficiently verify the consensus of a Tendermint blockchain. This forms the +basis of safe and efficient state synchronization for new network nodes and +inter-blockchain communication (where a light client of one Tendermint instance +runs in another chain's state machine). + +In a network that is expected to reliably punish validators for misbehavior +by slashing bonded stake and where the validator set changes +infrequently, clients can take advantage of this assumption to safely +synchronize a lite client without downloading the intervening headers. + +Light clients (and full nodes) operating in the Proof Of Stake context need a +trusted block height from a trusted source that is no older than 1 unbonding +window plus a configurable evidence submission synchrony bound. This is called “weak subjectivity”. + +Weak subjectivity is required in Proof of Stake blockchains because it is +costless for an attacker to buy up voting keys that are no longer bonded and +fork the network at some point in its prior history. See Vitalik’s post at +[Proof of Stake: How I Learned to Love Weak +Subjectivity](https://blog.ethereum.org/2014/11/25/proof-stake-learned-love-weak-subjectivity/). + +Currently, Tendermint provides a lite client implementation in the +[light](https://github.com/tendermint/tendermint/tree/main/light) package. This +lite client implements a bisection algorithm that tries to use a binary search +to find the minimum number of block headers where the validator set voting +power changes are less than < 1/3rd. This interface does not support weak +subjectivity at this time. The Cosmos SDK also does not support counterfactual +slashing, nor does the lite client have any capacity to report evidence making +these systems *theoretically unsafe*. + +NOTE: Tendermint provides a somewhat different (stronger) light client model +than Bitcoin under eclipse, since the eclipsing node(s) can only fool the light +client if they have two-thirds of the private keys from the last root-of-trust. + +## Decision + +### The Weak Subjectivity Interface + +Add the weak subjectivity interface for when a new light client connects to the +network or when a light client that has been offline for longer than the +unbonding period connects to the network. Specifically, the node needs to +initialize the following structure before syncing from user input: + +``` +type TrustOptions struct { + // Required: only trust commits up to this old. + // Should be equal to the unbonding period minus some delta for evidence reporting. + TrustPeriod time.Duration `json:"trust-period"` + + // Option 1: TrustHeight and TrustHash can both be provided + // to force the trusting of a particular height and hash. + // If the latest trusted height/hash is more recent, then this option is + // ignored. + TrustHeight int64 `json:"trust-height"` + TrustHash []byte `json:"trust-hash"` + + // Option 2: Callback can be set to implement a confirmation + // step if the trust store is uninitialized, or expired. + Callback func(height int64, hash []byte) error +} +``` + +The expectation is the user will get this information from a trusted source +like a validator, a friend, or a secure website. A more user friendly +solution with trust tradeoffs is that we establish an https based protocol with +a default end point that populates this information. Also an on-chain registry +of roots-of-trust (e.g. on the Cosmos Hub) seems likely in the future. + +### Linear Verification + +The linear verification algorithm requires downloading all headers +between the `TrustHeight` and the `LatestHeight`. The lite client downloads the +full header for the provided `TrustHeight` and then proceeds to download `N+1` +headers and applies the [Tendermint validation +rules](https://github.com/tendermint/tendermint/tree/main/spec/light-client/verification/README.md) +to each block. + +### Bisecting Verification + +Bisecting Verification is a more bandwidth and compute intensive mechanism that +in the most optimistic case requires a light client to only download two block +headers to come into synchronization. + +The bisection algorithm proceeds in the following fashion. The client downloads +and verifies the full block header for `TrustHeight` and then fetches +`LatestHeight` blocker header. The client then verifies the `LatestHeight` +header. Finally the client attempts to verify the `LatestHeight` header with +voting powers taken from `NextValidatorSet` in the `TrustHeight` header. This +verification will succeed if the validators from `TrustHeight` still have > 2/3 ++1 of voting power in the `LatestHeight`. If this succeeds, the client is fully +synchronized. If this fails, then following Bisection Algorithm should be +executed. + +The Client tries to download the block at the mid-point block between +`LatestHeight` and `TrustHeight` and attempts that same algorithm as above +using `MidPointHeight` instead of `LatestHeight` and a different threshold - +1/3 +1 of voting power for *non-adjacent headers*. In the case the of failure, +recursively perform the `MidPoint` verification until success then start over +with an updated `NextValidatorSet` and `TrustHeight`. + +If the client encounters a forged header, it should submit the header along +with some other intermediate headers as the evidence of misbehavior to other +full nodes. After that, it can retry the bisection using another full node. An +optimal client will cache trusted headers from the previous run to minimize +network usage. + +--- + +Check out the formal specification +[here](https://github.com/tendermint/tendermint/tree/main/spec/light-client). + +## Status + +Implemented + +## Consequences + +### Positive + +* light client which is safe to use (it can go offline, but not for too long) + +### Negative + +* complexity of bisection + +### Neutral + +* social consensus can be prone to errors (for cases where a new light client + joins a network or it has been offline for too long) diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-045-abci-evidence.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-045-abci-evidence.md new file mode 100644 index 00000000..65a0b688 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-045-abci-evidence.md @@ -0,0 +1,140 @@ +# ADR 45 - ABCI Evidence Handling + +## Changelog +* 21-09-2019: Initial draft + +## Context + +Evidence is a distinct component in a Tendermint block and has it's own reactor +for high priority gossipping. Currently, Tendermint supports only a single form of evidence, an explicit +equivocation, where a validator signs conflicting blocks at the same +height/round. It is detected in real-time in the consensus reactor, and gossiped +through the evidence reactor. Evidence can also be submitted through the RPC. + +Currently, Tendermint does not gracefully handle a fork on the main chain. +If a fork is detected, the node panics. At this point manual intervention and +social consensus are required to reconfigure. We'd like to do something more +graceful here, but that's for another day. + +It's possible to fool lite clients without there being a fork on the +main chain - so called Fork-Lite. See the +[fork accountability](https://github.com/tendermint/tendermint/blob/main/spec/light-client/accountability/README.md) +document for more details. For a sequential lite client, this can happen via +equivocation or amnesia attacks. For a skipping lite client this can also happen +via lunatic validator attacks. There must be some way for applications to punish +all forms of misbehavior. + +The essential question is whether Tendermint should manage the evidence +verification, or whether it should treat evidence more like a transaction (ie. +arbitrary bytes) and let the application handle it (including all the signature +checking). + +Currently, evidence verification is handled by Tendermint. Once committed, +[evidence is passed over +ABCI](https://github.com/tendermint/tendermint/blob/main/proto/tendermint/abci/types.proto#L354) +in BeginBlock in a reduced form that includes only +the type of evidence, its height and timestamp, the validator it's from, and the +total voting power of the validator set at the height. The app trusts Tendermint +to perform the evidence verification, as the ABCI evidence does not contain the +signatures and additional data for the app to verify itself. + +Arguments in favor of leaving evidence handling in Tendermint: + +1) Attacks on full nodes must be detectable by full nodes in real time, ie. within the consensus reactor. + So at the very least, any evidence involved in something that could fool a full + node must be handled natively by Tendermint as there would otherwise be no way + for the ABCI app to detect it (ie. we don't send all votes we receive during + consensus to the app ... ). + +2) Amensia attacks can not be easily detected - they require an interactive + protocol among all the validators to submit justification for their past + votes. Our best notion of [how to do this + currently](https://github.com/tendermint/tendermint/blob/c67154232ca8be8f5c21dff65d154127adc4f7bb/docs/spec/consensus/fork-detection.md) + is via a centralized + monitor service that is trusted for liveness to aggregate data from + current and past validators, but which produces a proof of misbehavior (ie. + via amnesia) that can be verified by anyone, including the blockchain. + Validators must submit all the votes they saw for the relevant consensus + height to justify their precommits. This is quite specific to the Tendermint + protocol and may change if the protocol is upgraded. Hence it would be awkward + to co-ordinate this from the app. + +3) Evidence gossipping is similar to tx gossipping, but it should be higher + priority. Since the mempool does not support any notion of priority yet, + evidence is gossipped through a distinct Evidence reactor. If we just treated + evidence like any other transaction, leaving it entirely to the application, + Tendermint would have no way to know how to prioritize it, unless/until we + significantly upgrade the mempool. Thus we would need to continue to treat evidence + distinctly and update the ABCI to either support sending Evidence through + CheckTx/DeliverTx, or to introduce new CheckEvidence/DeliverEvidence methods. + In either case we'd need to make more changes to ABCI then if Tendermint + handled things and we just added support for another evidence type that could be included + in BeginBlock. + +4) All ABCI application frameworks will benefit from most of the heavy lifting + being handled by Tendermint, rather than each of them needing to re-implement + all the evidence verification logic in each language. + +Arguments in favor of moving evidence handling to the application: + +5) Skipping lite clients require us to track the set of all validators that were + bonded over some period in case validators that are unbonding but still + slashable sign invalid headers to fool lite clients. The Cosmos-SDK + staking/slashing modules track this, as it's used for slashing. + Tendermint does not currently track this, though it does keep track of the + validator set at every height. This leans in favour of managing evidence in + the app to avoid redundantly managing the historical validator set data in + Tendermint + +6) Applications supporting cross-chain validation will be required to process + evidence from other chains. This data will come in the form of a transaction, + but it means the app will be required to have all the functionality to process + evidence, even if the evidence for its own chain is handled directly by + Tendermint. + +7) Evidence from lite clients may be large and constitute some form of DoS + vector against full nodes. Putting it in transactions allows it to engage the application's fee + mechanism to pay for cost of executions in the event the evidence is false. + This means the evidence submitter must be able to afford the fees for the + submission, but of course it should be refunded if the evidence is valid. + That said, the burden is mostly on full nodes, which don't necessarily benefit + from fees. + + +## Decision + +The above mostly seems to suggest that evidence detection belongs in Tendermint. +(5) does not impose particularly large obligations on Tendermint and (6) just +means the app can use Tendermint libraries. That said, (7) is potentially +cause for some concern, though it could still attack full nodes that weren't associated with validators +(ie. that don't benefit from fees). This could be handled out of band, for instance by +full nodes offering the light client service via payment channels or via some +other payment service. This can also be mitigated by banning client IPs if they +send bad data. Note the burden is on the client to actually send us a lot of +data in the first place. + +A separate ADR will describe how Tendermint will handle these new forms of +evidence, in terms of how it will engage the monitoring protocol described in +the [fork +detection](https://github.com/tendermint/tendermint/blob/c67154232ca8be8f5c21dff65d154127adc4f7bb/docs/spec/consensus/fork-detection.md) document, +and how it will track past validators and manage DoS issues. + +## Status + +Proposed. + +## Consequences + +### Positive + +- No real changes to ABCI +- Tendermint handles evidence for all apps + +### Neutral + +- Need to be careful about denial of service on the Tendermint RPC + +### Negative + +- Tendermint duplicates data by tracking all pubkeys that were validators during + the unbonding period diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-046-light-client-implementation.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-046-light-client-implementation.md new file mode 100644 index 00000000..15d77373 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-046-light-client-implementation.md @@ -0,0 +1,169 @@ +# ADR 046: Lite Client Implementation + +## Changelog +* 13-02-2020: Initial draft +* 26-02-2020: Cross-checking the first header +* 28-02-2020: Bisection algorithm details +* 31-03-2020: Verify signature got changed + +## Context + +A `Client` struct represents a light client, connected to a single blockchain. + +The user has an option to verify headers using `VerifyHeader` or +`VerifyHeaderAtHeight` or `Update` methods. The latter method downloads the +latest header from primary and compares it with the currently trusted one. + +```go +type Client interface { + // verify new headers + VerifyHeaderAtHeight(height int64, now time.Time) (*types.SignedHeader, error) + VerifyHeader(newHeader *types.SignedHeader, newVals *types.ValidatorSet, now time.Time) error + Update(now time.Time) (*types.SignedHeader, error) + + // get trusted headers & validators + TrustedHeader(height int64) (*types.SignedHeader, error) + TrustedValidatorSet(height int64) (valSet *types.ValidatorSet, heightUsed int64, err error) + LastTrustedHeight() (int64, error) + FirstTrustedHeight() (int64, error) + + // query configuration options + ChainID() string + Primary() provider.Provider + Witnesses() []provider.Provider + + Cleanup() error +} +``` + +A new light client can either be created from scratch (via `NewClient`) or +using the trusted store (via `NewClientFromTrustedStore`). When there's some +data in the trusted store and `NewClient` is called, the light client will a) +check if stored header is more recent b) optionally ask the user whenever it +should rollback (no confirmation required by default). + +```go +func NewClient( + chainID string, + trustOptions TrustOptions, + primary provider.Provider, + witnesses []provider.Provider, + trustedStore store.Store, + options ...Option) (*Client, error) { +``` + +`witnesses` as argument (as opposite to `Option`) is an intentional choice, +made to increase security by default. At least one witness is required, +although, right now, the light client does not check that primary != witness. +When cross-checking a new header with witnesses, minimum number of witnesses +required to respond: 1. Note the very first header (`TrustOptions.Hash`) is +also cross-checked with witnesses for additional security. + +Due to bisection algorithm nature, some headers might be skipped. If the light +client does not have a header for height `X` and `VerifyHeaderAtHeight(X)` or +`VerifyHeader(H#X)` methods are called, these will perform either a) backwards +verification from the latest header back to the header at height `X` or b) +bisection verification from the first stored header to the header at height `X`. + +`TrustedHeader`, `TrustedValidatorSet` only communicate with the trusted store. +If some header is not there, an error will be returned indicating that +verification is required. + +```go +type Provider interface { + ChainID() string + + SignedHeader(height int64) (*types.SignedHeader, error) + ValidatorSet(height int64) (*types.ValidatorSet, error) +} +``` + +Provider is a full node usually, but can be another light client. The above +interface is thin and can accommodate many implementations. + +If provider (primary or witness) becomes unavailable for a prolonged period of +time, it will be removed to ensure smooth operation. + +Both `Client` and providers expose chain ID to track if there are on the same +chain. Note, when chain upgrades or intentionally forks, chain ID changes. + +The light client stores headers & validators in the trusted store: + +```go +type Store interface { + SaveSignedHeaderAndValidatorSet(sh *types.SignedHeader, valSet *types.ValidatorSet) error + DeleteSignedHeaderAndValidatorSet(height int64) error + + SignedHeader(height int64) (*types.SignedHeader, error) + ValidatorSet(height int64) (*types.ValidatorSet, error) + + LastSignedHeaderHeight() (int64, error) + FirstSignedHeaderHeight() (int64, error) + + SignedHeaderAfter(height int64) (*types.SignedHeader, error) + + Prune(size uint16) error + + Size() uint16 +} +``` + +At the moment, the only implementation is the `db` store (wrapper around the KV +database, used in Tendermint). In the future, remote adapters are possible +(e.g. `Postgresql`). + +```go +func Verify( + chainID string, + trustedHeader *types.SignedHeader, // height=X + trustedVals *types.ValidatorSet, // height=X or height=X+1 + untrustedHeader *types.SignedHeader, // height=Y + untrustedVals *types.ValidatorSet, // height=Y + trustingPeriod time.Duration, + now time.Time, + maxClockDrift time.Duration, + trustLevel tmmath.Fraction) error { +``` + +`Verify` pure function is exposed for a header verification. It handles both +cases of adjacent and non-adjacent headers. In the former case, it compares the +hashes directly (2/3+ signed transition). Otherwise, it verifies 1/3+ +(`trustLevel`) of trusted validators are still present in new validators. + +While `Verify` function is certainly handy, `VerifyAdjacent` and +`VerifyNonAdjacent` should be used most often to avoid logic errors. + +### Bisection algorithm details + +Non-recursive bisection algorithm was implemented despite the spec containing +the recursive version. There are two major reasons: + +1) Constant memory consumption => no risk of getting OOM (Out-Of-Memory) exceptions; +2) Faster finality (see Fig. 1). + +_Fig. 1: Differences between recursive and non-recursive bisections_ + +![Fig. 1](./img/adr-046-fig1.png) + +Specification of the non-recursive bisection can be found +[here](https://github.com/tendermint/spec/blob/zm_non-recursive-verification/spec/consensus/light-client/non-recursive-verification.md). + +## Status + +Implemented + +## Consequences + +### Positive + +* single `Client` struct, which is easy to use +* flexible interfaces for header providers and trusted storage + +### Negative + +* `Verify` needs to be aligned with the current spec + +### Neutral + +* `Verify` function might be misused (called with non-adjacent headers in + incorrectly implemented sequential verification) diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-047-handling-evidence-from-light-client.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-047-handling-evidence-from-light-client.md new file mode 100644 index 00000000..b0cac65d --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-047-handling-evidence-from-light-client.md @@ -0,0 +1,254 @@ +# ADR 047: Handling evidence from light client + +## Changelog +* 18-02-2020: Initial draft +* 24-02-2020: Second version +* 13-04-2020: Add PotentialAmnesiaEvidence and a few remarks +* 31-07-2020: Remove PhantomValidatorEvidence +* 14-08-2020: Introduce light traces (listed now as an alternative approach) +* 20-08-2020: Light client produces evidence when detected instead of passing to full node +* 16-09-2020: Post-implementation revision +* 15-03-2020: Ammends for the case of a forward lunatic attack + +### Glossary of Terms + +- a `LightBlock` is the unit of data that a light client receives, verifies and stores. +It is composed of a validator set, commit and header all at the same height. +- a **Trace** is seen as an array of light blocks across a range of heights that were +created as a result of skipping verification. +- a **Provider** is a full node that a light client is connected to and serves the light +client signed headers and validator sets. +- `VerifySkipping` (sometimes known as bisection or verify non-adjacent) is a method the +light client uses to verify a target header from a trusted header. The process involves verifying +intermediate headers in between the two by making sure that 1/3 of the validators that signed +the trusted header also signed the untrusted one. +- **Light Bifurcation Point**: If the light client was to run `VerifySkipping` with two providers +(i.e. a primary and a witness), the bifurcation point is the height that the headers +from each of these providers are different yet valid. This signals that one of the providers +may be trying to fool the light client. + +## Context + +The bisection method of header verification used by the light client exposes +itself to a potential attack if any block within the light clients trusted period has +a malicious group of validators with power that exceeds the light clients trust level +(default is 1/3). To improve light client (and overall network) security, the light +client has a detector component that compares the verified header provided by the +primary against witness headers. This ADR outlines the process of mitigating attacks +on the light client by using witness nodes to cross reference with. + +## Alternative Approaches + +A previously discussed approach to handling evidence was to pass all the data that the +light client had witnessed when it had observed diverging headers for the full node to +process.This was known as a light trace and had the following structure: + +```go +type ConflictingHeadersTrace struct { + Headers []*types.SignedHeader +} +``` + +This approach has the advantage of not requiring as much processing on the light +client side in the event that an attack happens. Although, this is not a significant +difference as the light client would in any case have to validate all the headers +from both witness and primary. Using traces would consume a large amount of bandwidth +and adds a DDOS vector to the full node. + + +## Decision + +The light client will be divided into two components: a `Verifier` (either sequential or +skipping) and a `Detector` (see [Informal's Detector](https://github.com/informalsystems/tendermint-rs/blob/master/docs/spec/lightclient/detection/detection.md)) +. The detector will take the trace of headers from the primary and check it against all +witnesses. For a witness with a diverging header, the detector will first verify the header +by bisecting through all the heights defined by the trace that the primary provided. If valid, +the light client will trawl through both traces and find the point of bifurcation where it +can proceed to extract any evidence (as is discussed in detail later). + +Upon successfully detecting the evidence, the light client will send it to both primary and +witness before halting. It will not send evidence to other peers nor continue to verify the +primary's header against any other header. + + +## Detailed Design + +The verification process of the light client will start from a trusted header and use a bisectional +algorithm to verify up to a header at a given height. This becomes the verified header (does not +mean that it is trusted yet). All headers that were verified in between are cached and known as +intermediary headers and the entire array is sometimes referred to as a trace. + +The light client's detector then takes all the headers and runs the detect function. + +```golang +func (c *Client) detectDivergence(primaryTrace []*types.LightBlock, now time.Time) error +``` + +The function takes the last header it received, the target header and compares it against all the witnesses +it has through the following function: + +```golang +func (c *Client) compareNewHeaderWithWitness(errc chan error, h *types.SignedHeader, + witness provider.Provider, witnessIndex int) +``` + +The err channel is used to send back all the outcomes so that they can be processed in parallel. +Invalid headers result in dropping the witness, lack of response or not having the headers is ignored +just as headers that have the same hash. Headers, however, +of a different hash then trigger the detection process between the primary and that particular witness. + +This begins with verification of the witness's header via skipping verification which is run in tande +with locating the Light Bifurcation Point + +![](./img/light-client-detector.png) + +This is done with: + +```golang +func (c *Client) examineConflictingHeaderAgainstTrace( + trace []*types.LightBlock, + targetBlock *types.LightBlock, + source provider.Provider, + now time.Time, + ) ([]*types.LightBlock, *types.LightBlock, error) +``` + +which performs the following + +1. Checking that the trusted header is the same. Currently, they should not theoretically be different +because witnesses cannot be added and removed after the client is initialized. But we do this any way +as a sanity check. If this fails we have to drop the witness. + +2. Querying and verifying the witness's headers using bisection at the same heights of all the +intermediary headers of the primary (In the above example this is A, B, C, D, F, H). If bisection fails +or the witness stops responding then we can call the witness faulty and drop it. + +3. We eventually reach a verified header by the witness which is not the same as the intermediary header +(In the above example this is E). This is the point of bifurcation (This could also be the last header). + +4. There is a unique case where the trace that is being examined against has blocks that have a greater +height than the targetBlock. This can occur as part of a forward lunatic attack where the primary has +provided a light block that has a height greater than the head of the chain (see Appendix B). In this +case, the light client will verify the sources blocks up to the targetBlock and return the block in the +trace that is directly after the targetBlock in height as the `ConflictingBlock` + +This function then returns the trace of blocks from the witness node between the common header and the +divergent header of the primary as it is likely, as seen in the example to the right, that multiple +headers where required in order to verify the divergent one. This trace will +be used later (as is also described later in this document). + +![](./img/bifurcation-point.png) + +Now, that an attack has been detected, the light client must form evidence to prove it. There are +three types of attacks that either the primary or witness could have done to try fool the light client +into verifying the wrong header: Lunatic, Equivocation and Amnesia. As the consequence is the same and +the data required to prove it is also very similar, we bundle these attack styles together in a single +evidence: + +```golang +type LightClientAttackEvidence struct { + ConflictingBlock *LightBlock + CommonHeight int64 +} +``` + +The light client takes the stance of first suspecting the primary. Given the bifurcation point found +above, it takes the two divergent headers and compares whether the one from the primary is valid with +respect to the one from the witness. This is done by calling `isInvalidHeader()` which looks to see if +any one of the deterministically derived header fields differ from one another. This could be one of +`ValidatorsHash`, `NextValidatorsHash`, `ConsensusHash`, `AppHash`, and `LastResultsHash`. +In this case we know it's a Lunatic attack and to help the witness verify it we send the height +of the common header which is 1 in the example above or C in the example above that. If all these +hashes are the same then we can infer that it is either Equivocation or Amnesia. In this case we send +the height of the diverged headers because we know that the validator sets are the same, hence the +malicious nodes are still bonded at that height. In the example above, this is height 10 and the +example above that it is the height at E. + +The light client now has the evidence and broadcasts it to the witness. + +However, it could have been that the header the light client used from the witness against the primary +was forged, so before halting the light client swaps the process and thus suspects the witness and +uses the primary to create evidence. It calls `examineConflictingHeaderAgainstTrace` this time using +the witness trace found earlier. +If the primary was malicious it is likely that it will not respond but if it is innocent then the +light client will produce the same evidence but this time the conflicting +block will come from the witness node instead of the primary. The evidence is then formed and sent to +the primary node. + +This then ends the process and the verify function that was called at the start returns the error to +the user. + +For a detailed overview of how each of these three attacks can be conducted please refer to the +[fork accountability spec](https://github.com/tendermint/tendermint/blob/main/spec/consensus/light-client/accountability.md). + +## Full Node Verification + +When a full node receives evidence from the light client it will need to verify +it for itself before gossiping it to peers and trying to commit it on chain. This process is outlined + in [ADR-059](https://github.com/tendermint/tendermint/blob/main/docs/architecture/adr-059-evidence-composition-and-lifecycle.md). + +## Status + +Implemented + +## Consequences + +### Positive + +* Light client has increased security against Lunatic, Equivocation and Amnesia attacks. +* Do not need intermediate data structures to encapsulate the malicious behavior +* Generalized evidence makes the code simpler + +### Negative + +* Breaking change on the light client from versions 0.33.8 and below. Previous +versions will still send `ConflictingHeadersEvidence` but it won't be recognized +by the full node. Light clients will however still refuse the header and shut down. +* Amnesia attacks although detected, will not be able to be punished as it is not +clear from the current information which nodes behaved maliciously. +* Evidence module must handle both individual and grouped evidence. + +### Neutral + +## References + +* [Fork accountability spec](https://github.com/tendermint/tendermint/blob/main/spec/consensus/light-client/accountability.md) +* [ADR 056: Light client amnesia attacks](https://github.com/tendermint/tendermint/blob/main/docs/architecture/adr-056-light-client-amnesia-attacks.md) +* [ADR-059: Evidence Composition and Lifecycle](https://github.com/tendermint/tendermint/blob/main/docs/architecture/adr-059-evidence-composition-and-lifecycle.md) +* [Informal's Light Client Detector](https://github.com/informalsystems/tendermint-rs/blob/master/docs/spec/lightclient/detection/detection.md) + + +## Appendix A + +PhantomValidatorEvidence was used to capture when a validator that was still staked +(i.e. within the bonded period) but was not in the current validator set had voted for a block. + +In later discussions it was argued that although possible to keep phantom validator +evidence, any case a phantom validator that could have the capacity to be involved +in fooling a light client would have to be aided by 1/3+ lunatic validators. + +It would also be very unlikely that the new validators injected by the lunatic attack +would be validators that currently still have something staked. + +Not only this but there was a large degree of extra computation required in storing all +the currently staked validators that could possibly fall into the group of being +a phantom validator. Given this, it was removed. + +## Appendix B + +A unique flavor of lunatic attack is a forward lunatic attack. This is where a malicious +node provides a header with a height greater than the height of the blockchain. Thus there +are no witnesses capable of rebutting the malicious header. Such an attack will also +require an accomplice, i.e. at least one other witness to also return the same forged header. +Although such attacks can be any arbitrary height ahead, they must still remain within the +clock drift of the light clients real time. Therefore, to detect such an attack, a light +client will wait for a time + +``` +2 * MAX_CLOCK_DRIFT + LAG +``` + +for a witness to provide the latest block it has. Given the time constraints, if the witness +is operating at the head of the blockchain, it will have a header with an earlier height but +a later timestamp. This can be used to prove that the primary has submitted a lunatic header +which violates monotonically increasing time. diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-050-improved-trusted-peering.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-050-improved-trusted-peering.md new file mode 100644 index 00000000..d079e67b --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-050-improved-trusted-peering.md @@ -0,0 +1,58 @@ +# ADR 50: Improved Trusted Peering + +## Changelog +* 22-10-2019: Initial draft +* 05-11-2019: Modify `maximum-dial-period` to `persistent-peers-max-dial-period` + +## Context + +When `max-num-inbound-peers` or `max-num-outbound-peers` of a node is reached, the node cannot spare more slots to any peer +by inbound or outbound. Therefore, after a certain period of disconnection, any important peering can be lost indefinitely +because all slots are consumed by other peers, and the node stops trying to dial the peer anymore. + +This is happening because of two reasons, exponential backoff and absence of unconditional peering feature for trusted peers. + + +## Decision + +We would like to suggest solving this problem by introducing two parameters in `config.toml`, `unconditional-peer-ids` and +`persistent-peers-max-dial-period`. + +1) `unconditional-peer-ids` + +A node operator inputs list of ids of peers which are allowed to be connected by both inbound or outbound regardless of +`max-num-inbound-peers` or `max-num-outbound-peers` of user's node reached or not. + +2) `persistent-peers-max-dial-period` + +Terms between each dial to each persistent peer will not exceed `persistent-peers-max-dial-period` during exponential backoff. +Therefore, `dial-period` = min(`persistent-peers-max-dial-period`, `exponential-backoff-dial-period`) + +Alternative approach + +Persistent-peers is only for outbound, therefore it is not enough to cover the full utility of `unconditional-peer-ids`. +@creamers158(https://github.com/Creamers158) suggested putting id-only items into persistent-peers to be handled as +`unconditional-peer-ids`, but it needs very complicated struct exception for different structure of items in persistent-peers. +Therefore we decided to have `unconditional-peer-ids` to independently cover this use-case. + +## Status + +Proposed + +## Consequences + +### Positive + +A node operator can configure two new parameters in `config.toml` so that he/she can assure that tendermint will allow connections +from/to peers in `unconditional-peer-ids`. Also he/she can assure that every persistent peer will be dialed at least once in every +`persistent-peers-max-dial-period` term. It achieves more stable and persistent peering for trusted peers. + +### Negative + +The new feature introduces two new parameters in `config.toml` which needs explanation for node operators. + +### Neutral + +## References + +* two p2p feature enhancement proposal(https://github.com/tendermint/tendermint/issues/4053) diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-051-double-signing-risk-reduction.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-051-double-signing-risk-reduction.md new file mode 100644 index 00000000..2bf8db73 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-051-double-signing-risk-reduction.md @@ -0,0 +1,53 @@ +# ADR 051: Double Signing Risk Reduction + +## Changelog + +* 27-11-2019: Initial draft +* 13-01-2020: Separate into 2 ADR, This ADR will only cover Double signing Protection and ADR-052 handle Tendermint Mode +* 22-01-2020: change the title from "Double signing Protection" to "Double Signing Risk Reduction" + +## Context + +To provide a risk reduction method for double signing incidents mistakenly executed by validators +- Validators often mistakenly run duplicated validators to cause double-signing incident +- This proposed feature is to reduce the risk of mistaken double-signing incident by checking recent N blocks before voting begins +- When we think of such serious impact on double-signing incident, it is very reasonable to have multiple risk reduction algorithm built in node daemon + +## Decision + +We would like to suggest a double signing risk reduction method. + +- Methodology : query recent consensus results to find out whether node's consensus key is used on consensus recently or not +- When to check + - When the state machine starts `ConsensusReactor` after fully synced + - When the node is validator ( with privValidator ) + - When `cs.config.DoubleSignCheckHeight > 0` +- How to check + 1. When a validator is transformed from syncing status to fully synced status, the state machine check recent N blocks (`latest_height - double_sign_check_height`) to find out whether there exists consensus votes using the validator's consensus key + 2. If there exists votes from the validator's consensus key, exit state machine program +- Configuration + - We would like to suggest by introducing `double_sign_check_height` parameter in `config.toml` and cli, how many blocks state machine looks back to check votes + - `double_sign_check_height = {{ .Consensus.DoubleSignCheckHeight }}` in `config.toml` + - `tendermint node --consensus.double_sign_check_height` in cli + - State machine ignore checking procedure when `double_sign_check_height == 0` + +## Status + +Implemented + +## Consequences + +### Positive + +- Validators can avoid double signing incident by mistakes. (eg. If another validator node is voting on consensus, starting new validator node with same consensus key will cause panic stop of the state machine because consensus votes with the consensus key are found in recent blocks) +- We expect this method will prevent majority of double signing incident by mistakes. + +### Negative + +- When the risk reduction method is on, restarting a validator node will panic because the node itself voted on consensus with the same consensus key. So, validators should stop the state machine, wait for some blocks, and then restart the state machine to avoid panic stop. + +### Neutral + +## References + +- Issue [#4059](https://github.com/tendermint/tendermint/issues/4059) : double-signing protection diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-052-tendermint-mode.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-052-tendermint-mode.md new file mode 100644 index 00000000..04f3eb69 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-052-tendermint-mode.md @@ -0,0 +1,85 @@ +# ADR 052: Tendermint Mode + +## Changelog + +* 27-11-2019: Initial draft from ADR-051 +* 13-01-2020: Separate ADR Tendermint Mode from ADR-051 +* 29-03-2021: Update info regarding defaults + +## Context + +- Full mode: full mode does not have the capability to become a validator. +- Validator mode : this mode is exactly same as existing state machine behavior. sync without voting on consensus, and participate consensus when fully synced +- Seed mode : lightweight seed node maintaining an address book, p2p like [TenderSeed](https://gitlab.com/polychainlabs/tenderseed) + +## Decision + +We would like to suggest a simple Tendermint mode abstraction. These modes will live under one binary, and when initializing a node the user will be able to specify which node they would like to create. + +- Which reactor, component to include for each node + - full + - switch, transport + - reactors + - mempool + - consensus + - evidence + - blockchain + - p2p/pex + - statesync + - rpc (safe connections only) + - *~~no privValidator(priv_validator_key.json, priv_validator_state.json)~~* + - validator + - switch, transport + - reactors + - mempool + - consensus + - evidence + - blockchain + - p2p/pex + - statesync + - rpc (safe connections only) + - with privValidator(priv_validator_key.json, priv_validator_state.json) + - seed + - switch, transport + - reactor + - p2p/pex +- Configuration, cli command + - We would like to suggest by introducing `mode` parameter in `config.toml` and cli + - `mode = "{{ .BaseConfig.Mode }}"` in `config.toml` + - `tendermint start --mode validator` in cli + - full | validator | seednode + - There will be no default. Users will need to specify when they run `tendermint init` +- RPC modification + - `host:26657/status` + - return empty `validator_info` when in full mode + - no rpc server in seednode +- Where to modify in codebase + - Add switch for `config.Mode` on `node/node.go:DefaultNewNode` + - If `config.Mode==validator`, call default `NewNode` (current logic) + - If `config.Mode==full`, call `NewNode` with `nil` `privValidator` (do not load or generation) + - Need to add exception routine for `nil` `privValidator` to related functions + - If `config.Mode==seed`, call `NewSeedNode` (seed node version of `node/node.go:NewNode`) + - Need to add exception routine for `nil` `reactor`, `component` to related functions + +## Status + +Implemented + +## Consequences + +### Positive + +- Node operators can choose mode when they run state machine according to the purpose of the node. +- Mode can prevent mistakes because users have to specify which mode they want to run via flag. (eg. If a user want to run a validator node, she/he should explicitly write down validator as mode) +- Different mode needs different reactors, resulting in efficient resource usage. + +### Negative + +- Users need to study how each mode operate and which capability it has. + +### Neutral + +## References + +- Issue [#2237](https://github.com/tendermint/tendermint/issues/2237) : Tendermint "mode" +- [TenderSeed](https://gitlab.com/polychainlabs/tenderseed) : A lightweight Tendermint Seed Node. diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-053-state-sync-prototype.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-053-state-sync-prototype.md new file mode 100644 index 00000000..2d8c37ad --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-053-state-sync-prototype.md @@ -0,0 +1,254 @@ +# ADR 053: State Sync Prototype + +State sync is now [merged](https://github.com/tendermint/tendermint/pull/4705). Up-to-date ABCI documentation is [available](https://github.com/tendermint/spec/pull/90), refer to it rather than this ADR for details. + +This ADR outlines the plan for an initial state sync prototype, and is subject to change as we gain feedback and experience. It builds on discussions and findings in [ADR-042](./adr-042-state-sync.md), see that for background information. + +## Changelog + +* 2020-01-28: Initial draft (Erik Grinaker) + +* 2020-02-18: Updates after initial prototype (Erik Grinaker) + * ABCI: added missing `reason` fields. + * ABCI: used 32-bit 1-based chunk indexes (was 64-bit 0-based). + * ABCI: moved `RequestApplySnapshotChunk.chain_hash` to `RequestOfferSnapshot.app_hash`. + * Gaia: snapshots must include node versions as well, both for inner and leaf nodes. + * Added experimental prototype info. + * Added open questions and implementation plan. + +* 2020-03-29: Strengthened and simplified ABCI interface (Erik Grinaker) + * ABCI: replaced `chunks` with `chunk_hashes` in `Snapshot`. + * ABCI: removed `SnapshotChunk` message. + * ABCI: renamed `GetSnapshotChunk` to `LoadSnapshotChunk`. + * ABCI: chunks are now exchanged simply as `bytes`. + * ABCI: chunks are now 0-indexed, for parity with `chunk_hashes` array. + * Reduced maximum chunk size to 16 MB, and increased snapshot message size to 4 MB. + +* 2020-04-29: Update with final released ABCI interface (Erik Grinaker) + +## Context + +State sync will allow a new node to receive a snapshot of the application state without downloading blocks or going through consensus. This bootstraps the node significantly faster than the current fast sync system, which replays all historical blocks. + +Background discussions and justifications are detailed in [ADR-042](./adr-042-state-sync.md). Its recommendations can be summarized as: + +* The application periodically takes full state snapshots (i.e. eager snapshots). + +* The application splits snapshots into smaller chunks that can be individually verified against a chain app hash. + +* Tendermint uses the light client to obtain a trusted chain app hash for verification. + +* Tendermint discovers and downloads snapshot chunks in parallel from multiple peers, and passes them to the application via ABCI to be applied and verified against the chain app hash. + +* Historical blocks are not backfilled, so state synced nodes will have a truncated block history. + +## Tendermint Proposal + +This describes the snapshot/restore process seen from Tendermint. The interface is kept as small and general as possible to give applications maximum flexibility. + +### Snapshot Data Structure + +A node can have multiple snapshots taken at various heights. Snapshots can be taken in different application-specified formats (e.g. MessagePack as format `1` and Protobuf as format `2`, or similarly for schema versioning). Each snapshot consists of multiple chunks containing the actual state data, for parallel downloads and reduced memory usage. + +```proto +message Snapshot { + uint64 height = 1; // The height at which the snapshot was taken + uint32 format = 2; // The application-specific snapshot format + uint32 chunks = 3; // Number of chunks in the snapshot + bytes hash = 4; // Arbitrary snapshot hash - should be equal only for identical snapshots + bytes metadata = 5; // Arbitrary application metadata +} +``` + +Chunks are exchanged simply as `bytes`, and cannot be larger than 16 MB. `Snapshot` messages should be less than 4 MB. + +### ABCI Interface + +```proto +// Lists available snapshots +message RequestListSnapshots {} + +message ResponseListSnapshots { + repeated Snapshot snapshots = 1; +} + +// Offers a snapshot to the application +message RequestOfferSnapshot { + Snapshot snapshot = 1; // snapshot offered by peers + bytes app_hash = 2; // light client-verified app hash for snapshot height + } + +message ResponseOfferSnapshot { + Result result = 1; + + enum Result { + accept = 0; // Snapshot accepted, apply chunks + abort = 1; // Abort all snapshot restoration + reject = 2; // Reject this specific snapshot, and try a different one + reject_format = 3; // Reject all snapshots of this format, and try a different one + reject_sender = 4; // Reject all snapshots from the sender(s), and try a different one + } +} + +// Loads a snapshot chunk +message RequestLoadSnapshotChunk { + uint64 height = 1; + uint32 format = 2; + uint32 chunk = 3; // Zero-indexed +} + +message ResponseLoadSnapshotChunk { + bytes chunk = 1; +} + +// Applies a snapshot chunk +message RequestApplySnapshotChunk { + uint32 index = 1; + bytes chunk = 2; + string sender = 3; + } + +message ResponseApplySnapshotChunk { + Result result = 1; + repeated uint32 refetch_chunks = 2; // Chunks to refetch and reapply (regardless of result) + repeated string reject_senders = 3; // Chunk senders to reject and ban (regardless of result) + + enum Result { + accept = 0; // Chunk successfully accepted + abort = 1; // Abort all snapshot restoration + retry = 2; // Retry chunk, combine with refetch and reject as appropriate + retry_snapshot = 3; // Retry snapshot, combine with refetch and reject as appropriate + reject_snapshot = 4; // Reject this snapshot, try a different one but keep sender rejections + } +} +``` + +### Taking Snapshots + +Tendermint is not aware of the snapshotting process at all, it is entirely an application concern. The following guarantees must be provided: + +* **Periodic:** snapshots must be taken periodically, not on-demand, for faster restores, lower load, and less DoS risk. + +* **Deterministic:** snapshots must be deterministic, and identical across all nodes - typically by taking a snapshot at given height intervals. + +* **Consistent:** snapshots must be consistent, i.e. not affected by concurrent writes - typically by using a data store that supports versioning and/or snapshot isolation. + +* **Asynchronous:** snapshots must be asynchronous, i.e. not halt block processing and state transitions. + +* **Chunked:** snapshots must be split into chunks of reasonable size (on the order of megabytes), and each chunk must be verifiable against the chain app hash. + +* **Garbage collected:** snapshots must be garbage collected periodically. + +### Restoring Snapshots + +Nodes should have options for enabling state sync and/or fast sync, and be provided a trusted header hash for the light client. + +When starting an empty node with state sync and fast sync enabled, snapshots are restored as follows: + +1. The node checks that it is empty, i.e. that it has no state nor blocks. + +2. The node contacts the given seeds to discover peers. + +3. The node contacts a set of full nodes, and verifies the trusted block header using the given hash via the light client. + +4. The node requests available snapshots via P2P from peers, via `RequestListSnapshots`. Peers will return the 10 most recent snapshots, one message per snapshot. + +5. The node aggregates snapshots from multiple peers, ordered by height and format (in reverse). If there are mismatches between different snapshots, the one hosted by the largest amount of peers is chosen. The node iterates over all snapshots in reverse order by height and format until it finds one that satisfies all of the following conditions: + + * The snapshot height's block is considered trustworthy by the light client (i.e. snapshot height is greater than trusted header and within unbonding period of the latest trustworthy block). + + * The snapshot's height or format hasn't been explicitly rejected by an earlier `RequestOfferSnapshot`. + + * The application accepts the `RequestOfferSnapshot` call. + +6. The node downloads chunks in parallel from multiple peers, via `RequestLoadSnapshotChunk`. Chunk messages cannot exceed 16 MB. + +7. The node passes chunks sequentially to the app via `RequestApplySnapshotChunk`. + +8. Once all chunks have been applied, the node compares the app hash to the chain app hash, and if they do not match it either errors or discards the state and starts over. + +9. The node switches to fast sync to catch up blocks that were committed while restoring the snapshot. + +10. The node switches to normal consensus mode. + +## Gaia Proposal + +This describes the snapshot process seen from Gaia, using format version `1`. The serialization format is unspecified, but likely to be compressed Amino or Protobuf. + +### Snapshot Metadata + +In the initial version there is no snapshot metadata, so it is set to an empty byte buffer. + +Once all chunks have been successfully built, snapshot metadata should be stored in a database and served via `RequestListSnapshots`. + +### Snapshot Chunk Format + +The Gaia data structure consists of a set of named IAVL trees. A root hash is constructed by taking the root hashes of each of the IAVL trees, then constructing a Merkle tree of the sorted name/hash map. + +IAVL trees are versioned, but a snapshot only contains the version relevant for the snapshot height. All historical versions are ignored. + +IAVL trees are insertion-order dependent, so key/value pairs must be set in an appropriate insertion order to produce the same tree branching structure. This insertion order can be found by doing a breadth-first scan of all nodes (including inner nodes) and collecting unique keys in order. However, the node hash also depends on the node's version, so snapshots must contain the inner nodes' version numbers as well. + +For the initial prototype, each chunk consists of a complete dump of all node data for all nodes in an entire IAVL tree. Thus the number of chunks equals the number of persistent stores in Gaia. No incremental verification of chunks is done, only a final app hash comparison at the end of the snapshot restoration. + +For a production version, it should be sufficient to store key/value/version for all nodes (leaf and inner) in insertion order, chunked in some appropriate way. If per-chunk verification is required, the chunk must also contain enough information to reconstruct the Merkle proofs all the way up to the root of the multistore, e.g. by storing a complete subtree's key/value/version data plus Merkle hashes of all other branches up to the multistore root. The exact approach will depend on tradeoffs between size, time, and verification. IAVL RangeProofs are not recommended, since these include redundant data such as proofs for intermediate and leaf nodes that can be derived from the above data. + +Chunks should be built greedily by collecting node data up to some size limit (e.g. 10 MB) and serializing it. Chunk data is stored in the file system as `snapshots///`, and a SHA-256 checksum is stored along with the snapshot metadata. + +### Snapshot Scheduling + +Snapshots should be taken at some configurable height interval, e.g. every 1000 blocks. All nodes should preferably have the same snapshot schedule, such that all nodes can serve chunks for a given snapshot. + +Taking consistent snapshots of IAVL trees is greatly simplified by them being versioned: simply snapshot the version that corresponds to the snapshot height, while concurrent writes create new versions. IAVL pruning must not prune a version that is being snapshotted. + +Snapshots must also be garbage collected after some configurable time, e.g. by keeping the latest `n` snapshots. + +## Resolved Questions + +* Is it OK for state-synced nodes to not have historical blocks nor historical IAVL versions? + + > Yes, this is as intended. Maybe backfill blocks later. + +* Do we need incremental chunk verification for first version? + + > No, we'll start simple. Can add chunk verification via a new snapshot format without any breaking changes in Tendermint. For adversarial conditions, maybe consider support for whitelisting peers to download chunks from. + +* Should the snapshot ABCI interface be a separate optional ABCI service, or mandatory? + + > Mandatory, to keep things simple for now. It will therefore be a breaking change and push the release. For apps using the Cosmos SDK, we can provide a default implementation that does not serve snapshots and errors when trying to apply them. + +* How can we make sure `ListSnapshots` data is valid? An adversary can provide fake/invalid snapshots to DoS peers. + + > For now, just pick snapshots that are available on a large number of peers. Maybe support whitelisting. We may consider e.g. placing snapshot manifests on the blockchain later. + +* Should we punish nodes that provide invalid snapshots? How? + + > No, these are full nodes not validators, so we can't punish them. Just disconnect from them and ignore them. + +* Should we call these snapshots? The SDK already uses the term "snapshot" for `PruningOptions.SnapshotEvery`, and state sync will introduce additional SDK options for snapshot scheduling and pruning that are not related to IAVL snapshotting or pruning. + + > Yes. Hopefully these concepts are distinct enough that we can refer to state sync snapshots and IAVL snapshots without too much confusion. + +* Should we store snapshot and chunk metadata in a database? Can we use the database for chunks? + + > As a first approach, store metadata in a database and chunks in the filesystem. + +* Should a snapshot at height H be taken before or after the block at H is processed? E.g. RPC `/commit` returns app_hash after _previous_ height, i.e. _before_ current height. + + > After commit. + +* Do we need to support all versions of blockchain reactor (i.e. fast sync)? + + > We should remove the v1 reactor completely once v2 has stabilized. + +* Should `ListSnapshots` be a streaming API instead of a request/response API? + + > No, just use a max message size. + +## Status + +Implemented + +## References + +* [ADR-042](./adr-042-state-sync.md) and its references diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-054-crypto-encoding-2.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-054-crypto-encoding-2.md new file mode 100644 index 00000000..e58681d1 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-054-crypto-encoding-2.md @@ -0,0 +1,71 @@ +# ADR 054: Crypto encoding (part 2) + +## Changelog + +2020-2-27: Created +2020-4-16: Update + +## Context + +Amino has been a pain point of many users in the ecosystem. While Tendermint does not suffer greatly from the performance degradation introduced by amino, we are making an effort in moving the encoding format to a widely adopted format, [Protocol Buffers](https://developers.google.com/protocol-buffers). With this migration a new standard is needed for the encoding of keys. This will cause ecosystem wide breaking changes. + +Currently amino encodes keys as ` `. + +## Decision + +Previously Tendermint defined all the key types for use in Tendermint and the Cosmos-SDK. Going forward the Cosmos-SDK will define its own protobuf type for keys. This will allow Tendermint to only define the keys that are being used in the codebase (ed25519). +There is the the opportunity to only define the usage of ed25519 (`bytes`) and not have it be a `oneof`, but this would mean that the `oneof` work is only being postponed to a later date. When using the `oneof` protobuf type we will have to manually switch over the possible key types and then pass them to the interface which is needed. + +The approach that will be taken to minimize headaches for users is one where all encoding of keys will shift to protobuf and where amino encoding is relied on, there will be custom marshal and unmarshal functions. + +Protobuf messages: + +```proto +message PubKey { + oneof key { + bytes ed25519 = 1; + } + +message PrivKey { + oneof sum { + bytes ed25519 = 1; + } +} +``` + +> Note: The places where backwards compatibility is needed is still unclear. + +All modules currently do not rely on amino encoded bytes and keys are not amino encoded for genesis, therefore a hardfork upgrade is what will be needed to adopt these changes. + +This work will be broken out into a few PRs, this work will be merged into a proto-breakage branch, all PRs will be reviewed prior to being merged: + +1. Encoding of keys to protobuf and protobuf messages +2. Move Tendermint types to protobuf, mainly the ones that are being encoded. +3. Go one by one through the reactors and transition amino encoded messages to protobuf. +4. Test with cosmos-sdk and/or testnets repo. + +## Status + +Implemented + +## Consequences + +- Move keys to protobuf encoding, where backwards compatibility is needed, amino marshal and unmarshal functions will be used. + +### Positive + +- Protocol Buffer encoding will not change going forward. +- Removing amino overhead from keys will help with the KSM. +- Have a large ecosystem of supported languages. + +### Negative + +- Hardfork is required to integrate this into running chains. + +### Neutral + +## References + +> Are there any relevant PR comments, issues that led up to this, or articles referenced for why we made the given design choice? If so link them here! + +- {reference link} diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-055-protobuf-design.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-055-protobuf-design.md new file mode 100644 index 00000000..ab2f7528 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-055-protobuf-design.md @@ -0,0 +1,61 @@ +# ADR 055: Protobuf Design + +## Changelog + +- 2020-4-15: Created (@marbar3778) +- 2020-6-18: Updated (@marbar3778) + +## Context + +Currently we use [go-amino](https://github.com/tendermint/go-amino) throughout Tendermint. Amino is not being maintained anymore (April 15, 2020) by the Tendermint team and has been found to have issues: + +- https://github.com/tendermint/go-amino/issues/286 +- https://github.com/tendermint/go-amino/issues/230 +- https://github.com/tendermint/go-amino/issues/121 + +These are a few of the known issues that users could run into. + +Amino enables quick prototyping and development of features. While this is nice, amino does not provide the performance and developer convenience that is expected. For Tendermint to see wider adoption as a BFT protocol engine a transition to an adopted encoding format is needed. Below are some possible options that can be explored. + +There are a few options to pick from: + +- `Protobuf`: Protocol buffers are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler. It is supported in countless languages and has been proven in production for many years. + +- `FlatBuffers`: FlatBuffers is an efficient cross platform serialization library. Flatbuffers are more efficient than Protobuf due to the fast that there is no parsing/unpacking to a second representation. FlatBuffers has been tested and used in production but is not widely adopted. + +- `CapnProto`: Cap’n Proto is an insanely fast data interchange format and capability-based RPC system. Cap'n Proto does not have a encoding/decoding step. It has not seen wide adoption throughout the industry. + +- @erikgrinaker - https://github.com/tendermint/tendermint/pull/4623#discussion_r401163501 + ``` + Cap'n'Proto is awesome. It was written by one of the original Protobuf developers to fix some of its issues, and supports e.g. random access to process huge messages without loading them into memory and an (opt-in) canonical form which would be very useful when determinism is needed (e.g. in the state machine). That said, I suspect Protobuf is the better choice due to wider adoption, although it makes me kind of sad since Cap'n'Proto is technically better. + ``` + +## Decision + +Transition Tendermint to Protobuf because of its performance and tooling. The Ecosystem behind Protobuf is vast and has outstanding [support for many languages](https://developers.google.com/protocol-buffers/docs/tutorials). + +We will be making this possible by keeping the current types in there current form (handwritten) and creating a `/proto` directory in which all the `.proto` files will live. Where encoding is needed, on disk and over the wire, we will call util functions that will transition the types from handwritten go types to protobuf generated types. This is inline with the recommended file structure from [buf](https://buf.build). You can find more information on this file structure [here](https://buf.build/docs/lint-checkers#file_layout). + +By going with this design we will enable future changes to types and allow for a more modular codebase. + +## Status + +Implemented + +## Consequences + +### Positive + +- Allows for modular types in the future +- Less refactoring +- Allows the proto files to be pulled into the spec repo in the future. +- Performance +- Tooling & support in multiple languages + +### Negative + +- When a developer is updating a type they need to make sure to update the proto type as well + +### Neutral + +## References diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-056-light-client-amnesia-attacks.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-056-light-client-amnesia-attacks.md new file mode 100644 index 00000000..68ac0f70 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-056-light-client-amnesia-attacks.md @@ -0,0 +1,170 @@ +# ADR 056: Light client amnesia attacks + +## Changelog + +- 02.04.20: Initial Draft +- 06.04.20: Second Draft +- 10.06.20: Post Implementation Revision +- 19.08.20: Short Term Amnesia Alteration +- 01.10.20: Status of Amnesia for 0.34 + +## Context + +Whilst most created evidence of malicious behavior is self evident such that any individual can verify them independently there are types of evidence, known collectively as global evidence, that require further collaboration from the network in order to accumulate enough information to create evidence that is individually verifiable and can therefore be processed through consensus. [Fork Accountability](https://github.com/tendermint/tendermint/blob/main/spec/consensus/light-client/accountability.md) has been coined to describe the entire process of detection, proving and punishing of malicious behavior. This ADR addresses specifically what a light client amnesia attack is and how it can be proven and the current decision around handling light client amnesia attacks. For information on evidence handling by the light client, it is recommended to read [ADR 47](https://github.com/tendermint/tendermint/blob/main/docs/architecture/adr-047-handling-evidence-from-light-client.md). + +### Amnesia Attack + +The schematic below explains a scenario where an amnesia attack can occur such that two sets of honest nodes, C1 and C2, commit different blocks. + +![](./img/tm-amnesia-attack.png) + +1. C1 and F send PREVOTE messages for block A. +2. C1 sends PRECOMMIT for round 1 for block A. +3. A new round is started, C2 and F send PREVOTE messages for a different block B. +4. C2 and F then send PRECOMMIT messages for block B. +5. F later on creates PRECOMMITS for block A and combines it with those from C1 to form a block + + +This forged block can then be used to fool light clients trying to verify it. It must be stressed that there are a few more hurdles or dimensions to the attack to consider.For a more detailed walkthrough refer to Appendix A. + +## Decision + +The decision surrounding amnesia attacks has both a short term and long term component. In the long term, a more sturdy protocol will need to be fleshed out and implemented. There is already draft documents outlining what such a protocol would look like and the resources it would require (see references). Prior revisions however outlined a protocol which had been implemented (See Appendix B). It was agreed that it still required greater consideration and review given it's importance. It was therefore discussed, with the limited time frame set before 0.34, whether the protocol should be completely removed or if there should remain some logic in handling the aforementioned scenarios. + +The latter of the two options meant storing a record of all votes in any height with which there was more than one round. This information would then be accessible for applications if they wanted to perform some off-chain verification and punishment. + +In summary, this seemed like too much to ask of the application to implement only on a temporary basis, whilst not having the domain specific knowledge and considering such a difficult and unlikely attack. Therefore the short term decision is to identify when the attack has occurred and implement the detector algorithm highlighted in [ADR 47](https://github.com/tendermint/tendermint/blob/main/docs/architecture/adr-047-handling-evidence-from-light-client.md) but to not implement any accountability protocol that would identify malicious validators and allow applications to punish them. This will hopefully change in the long term with the focus on eventually reaching a concrete and secure protocol with identifying and dealing with these attacks. + +## Implications + +- Light clients will still be able to detect amnesia attacks so long as the assumption of having at least one correct witness holds +- Light clients will gossip the attack to witnesses and halt thus failing to validate the incorrect block (and therefore not being fooled) +- Validators will propose and commit evidence of the amnesia attack on chain +- No evidence will be passed to the application indicting any malicious validators, thus meaning that no malicious validators will be punished for performing the attack +- If a light clients bubble of providers are all faulty the light client will falsely validate amnesia attacks as well as any other 1/3+ light client attack. + +## Status + +Implemented + +## Consequences + +### Positive + +Light clients are still able to prevent falsely validating a block. + +Already implemented. + +### Negative + +Light clients where all witnesses are faulty can be subject to an amnesia attack and verify a forged block that is not part of the chain. + +### Neutral + + +## References + +- [Fork accountability algorithm](https://docs.google.com/document/d/11ZhMsCj3y7zIZz4udO9l25xqb0kl7gmWqNpGVRzOeyY/edit) +- [Fork accountability spec](https://github.com/tendermint/tendermint/blob/main/spec/consensus/light-client/accountability.md) + +## Appendix A: Detailed Walkthrough of Performing a Light Client Amnesia Attack + +As the attacker, a prerequisite to this attack is first to observe or attempt to craft a block where a subset (less than ⅓) of correct validators sent precommit votes for a proposal in an earlier round and later received ⅔ prevotes for a different proposal thus changing their lock and correctly sending precommit votes (and later committing) for the proposal in the latter round. The second prerequisite is to have at least ⅓ validating power in that height (or enough voting power to have ⅔+ when combined with the precommits of the earlier round). + +To go back to how one may craft such a block, we begin with one of the validators in this cabal being the proposer. They propose a block with all the txs that they want to fool a light client with. The proposer then only relays this to the members of their cabal and a controlled subset of correct validators (less than ⅓). We will call ourselves f for faulty and c1 for this correct subset. + +Attackers need to rely on the assistance of some form of a network partition or on the nature of the sporadic voting to conjure their desired environment. The attackers need at least ⅓ of the validating power of the remaining correct validators, we shall denote this as c2, to not see ⅔ prevotes and thus not be locked on a block when it comes to the next round. If we have less than ⅓ remaining validators that don’t see this first proposal, then we will not have enough voting power to reach ⅔+ prevotes (the sum of f and c2) in the following round and thus change the lock of c1 such that we correctly commit the block in the latter round yet have enough precommits in the earlier round to fool the light client. Remember this is our desired scenario: to save all these precommit votes for a different (in this case earlier) proposed block. + +To try to break this down even further let’s go back to the first round. F sends c1 a proposal (and not c2), c1 in turn sends their prevotes to all whom they are connected to. This means that some will be received by c2. F then sends their prevotes just to c1. Now not all validators in c1 may be connected to each other, so perhaps some validators in c1 might not receive ⅔ (from their own cohort and from f) and thus not precommit. In other situations we may see a validator in c2 connected to all validators in c1. Therefore they too will receive ⅔ prevotes and thus precommit. We can conclude therefore that although targeting this c1 subset of validators, those that actually precommit may be somewhat different. The key is for the attackers to observe the n amount of precommits they need in round 1 where n is ⅔+ - f, whilst ensuring that n itself does not go over ⅓. If it does then less than ⅔ validators remain to be able to change the lock and commit the block in the later round. + +An extra dimension to this puzzle is the timeouts. Whilst c1 is relaying votes to its peers and these validators count closer towards the ⅔ threshold needed to send their precommit votes at any moment the timeout could be reached and thus the nodes will precommit nil and ignore any late prevote messages. + +This is all to say that such an attack is partly out of the attackers hands. All they can do is tweak the subset of validators that they first choose to gossip the proposal and modify the timings around when they send their prevotes until they reach the desired precondition: n precommits for an earlier proposal and ⅔ precommits for the later proposal. So this is up to the gods of non deterministic behavior to help them out with their plight. I’m not going to allocate the hours to calculate the probability but it could be in the magnitude of 1000’s of blocks trying to get this scenario before the precondition is met. + +Obviously, the probability becomes substantially higher as the cabal’s voting power nears ⅔. This is because both n decreases and there is greater tolerance to send prevotes to a greater amount of validators without going overboard and reaching the ⅓ precommit threshold in the first round which would mean they would have to try again. + +Once we’ve got our n, we can then forge the remaining signatures for that block (from the f) and bundle them all together and tada we have a forged signed header. + +Now we’ve done that, it’s time to find some light clients to fool. + +Also critical to this type of attack is that the light client that is connected to our nodes must request a light block at that specific height with which we forged this signed header but this shouldn’t be hard to do. To bring this back to a real context, say our faulty cabal, f, bought some groceries using atoms and then wanted to prove that they did, the grocery owner whips out their phone, runs the light client and f tells them the height they committed the transaction. + +An important note here is that because the validator sets are the same between the canonical and the forged block, this attack also works on light clients that verify sequentially. In fact, they are especially vulnerable because they currently don’t run the detector function afterwards. + +However, if our grocery owner verifies using the skipping algorithm, they will then run the detector and therefore they will compare with other witness nodes. Ideally for our attackers, if f has a lot of nodes exposing their rpc endpoints, then there is a chance that all the witnesses the light client has are faulty and thus we have a successful attack and the grocery owner has been fooled into handing f a few apples and carrots. + +However, there is a greater chance, especially if the light client is connected to quite a few other nodes that a divergence will be detected. The light client will figure out there was an amnesia attack and send the evidence to the witness to commit on chain. The grocery owner will see that verification failed and won't hand over the apples or carrots but also f won't be punished for their villainous behavior. This means that they can go over to the hairdressers and see if they can pull off the same stunt again. + +So this brings to the fore the current defenses that are in place. As long as there has not been a cabal of validators with greater than 1/3 power (or the trust level), the light clients verification algorithm will prevent any attempts to deceive it. Greater than this threshold and we rely on the detector as a second layer of defense to pick up on any attack. It's security is chiefly tied with the assumption that at least one of the witnesses is correct. If this fails then as illustrated above, the light client can be suceptible to amnesia (as well as equivocation and lunatic) attacks. + +The outstanding problem, if we indeed consider it big enough to be one, therefore lies in the incentivisation mechanism which is how f and other malicious validators are punished. This is decided by the application but it's up to Tendermint to identify them. With other forms of attacks the evidence lies in the precommits. But because an amnesia attack uses precommits from another round, which is information that is discarded by the consensus engine once the block is committed, it is difficult to understand which validators were in fact faulty. + +If we cast our minds back to what I previously wrote, part of an amnesia attack depends on getting n precommits from an earlier round. These are then bundled with the malicious validators' own signatures. This means that the light client nor full nodes are capable of distinguishing which of the signatures were correctly created as part of Tendermint consensus and which were forged later on. + +## Appendix B: Prior Amnesia Evidence Accountability Implementation + +As the distinction between these two attacks (amnesia and back to the past) can only be distinguished by confirming with all validators (to see if it is a full fork or a light fork), for the purpose of simplicity, these attacks will be treated as the same. + +Currently, the evidence reactor is used to simply broadcast and store evidence. The idea of creating a new reactor for the specific task of verifying these attacks was briefly discussed, but it is decided that the current evidence reactor will be extended. + +The process begins with a light client receiving conflicting headers (in the future this could also be a full node during fast sync or state sync), which it sends to a full node to analyze. As part of [evidence handling](https://github.com/tendermint/tendermint/blob/main/docs/architecture/adr-047-handling-evidence-from-light-client.md), this is extracted into potential amnesia evidence when the validator voted in more than one round for a different block. + +```golang +type PotentialAmnesiaEvidence struct { + VoteA *types.Vote + VoteB *types.Vote + + Heightstamp int64 +} +``` + +*NOTE: There had been an earlier notion towards batching evidence against the entire set of validators all together but this has given way to individual processing predominantly to maintain consistency with the other forms of evidence. A more extensive breakdown can be found [here](https://github.com/tendermint/tendermint/issues/4729)* + +The evidence will contain the precommit votes for a validator that voted for both rounds. If the validator voted in more than two rounds, then they will have multiple `PotentialAmnesiaEvidence` against them hence it is possible that there is multiple evidence for a validator in a single height but not for a single round. The votes should be all valid and the height and time that the infringement was made should be within: + +`MaxEvidenceAge - ProofTrialPeriod` + +This trial period will be discussed later. + +Returning to the event of an amnesia attack, if we were to examine the behavior of the honest nodes, C1 and C2, in the schematic, C2 will not PRECOMMIT an earlier round, but it is likely, if a node in C1 were to receive +2/3 PREVOTE's or PRECOMMIT's for a higher round, that it would remove the lock and PREVOTE and PRECOMMIT for the later round. Therefore, unfortunately it is not a case of simply punishing all nodes that have double voted in the `PotentialAmnesiaEvidence`. + +Instead we use the Proof of Lock Change (PoLC) referred to in the [consensus spec](https://github.com/tendermint/tendermint/blob/main/spec/consensus/consensus.md#terms). When an honest node votes again for a different block in a later round +(which will only occur in very rare cases), it will generate the PoLC and store it in the evidence reactor for a time equal to the `MaxEvidenceAge` + +```golang +type ProofOfLockChange struct { + Votes []*types.Vote + PubKey crypto.PubKey +} +``` + +This can be either evidence of +2/3 PREVOTES or PRECOMMITS (either warrants the honest node the right to vote) and is valid, among other checks, so long as the PRECOMMIT vote of the node in V2 came after all the votes in the `ProofOfLockChange` i.e. it received +2/3 votes for a block and then voted for that block thereafter (F is unable to prove this). + +In the event that an honest node receives `PotentialAmnesiaEvidence` it will first `ValidateBasic()` and `Verify()` it and then will check if it is among the suspected nodes in the evidence. If so, it will retrieve the `ProofOfLockChange` and combine it with `PotentialAmensiaEvidence` to form `AmensiaEvidence`. All honest nodes that are part of the indicted group will have a time, measured in blocks, equal to `ProofTrialPeriod`, the aforementioned evidence paramter, to gossip their `AmnesiaEvidence` with their `ProofOfLockChange` + +```golang +type AmnesiaEvidence struct { + *types.PotentialAmnesiaEvidence + Polc *types.ProofOfLockChange +} +``` + +If the node is not required to submit any proof than it will simply broadcast the `PotentialAmnesiaEvidence`, stamp the height that it received the evidence and begin to wait out the trial period. It will ignore other `PotentialAmnesiaEvidence` gossiped at the same height and round. + +If a node receives `AmnesiaEvidence` that contains a valid `ProofOfClockChange` it will add it to the evidence store and replace any PotentialAmnesiaEvidence of the same height and round. At this stage, an amnesia evidence with polc, it is ready to be submitted to the chin. If a node receives `AmnesiaEvidence` with an empty polc it will ignore it as each honest node will conduct their own trial period to be sure that time was given for any other honest nodes to respond. + +There can only be one `AmnesiaEvidence` and one `PotentialAmneisaEvidence` stored for each attack (i.e. for each height). + +When, `state.LastBlockHeight > PotentialAmnesiaEvidence.timestamp + ProofTrialPeriod`, nodes will upgrade the corresponding `PotentialAmnesiaEvidence` and attach an empty `ProofOfLockChange`. Then honest validators of the current validator set can begin proposing the block that contains the `AmnesiaEvidence`. + +*NOTE: Even before the evidence is proposed and committed, the off-chain process of gossiping valid evidence could be + enough for honest nodes to recognize the fork and halt.* + +Other validators will vote `nil` if: + +- The Amnesia Evidence is not valid +- The Amensia Evidence is not within their own trial period i.e. too soon. +- They don't have the Amnesia Evidence and it is has an empty polc (each validator needs to run their own trial period of the evidence) +- Is of an AmnesiaEvidence that has already been committed to the chain. + +Finally it is important to stress that the protocol of having a trial period addresses attacks where a validator voted again for a different block at a later round and time. In the event, however, that the validator voted for an earlier round after voting for a later round i.e. `VoteA.Timestamp < VoteB.Timestamp && VoteA.Round > VoteB.Round` then this action is inexcusable and can be punished immediately without the need of a trial period. In this case, PotentialAmnesiaEvidence will be instantly upgraded to AmnesiaEvidence. diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-057-RPC.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-057-RPC.md new file mode 100644 index 00000000..5e7c9f1d --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-057-RPC.md @@ -0,0 +1,90 @@ +# ADR 057: RPC + +## Changelog + +- 19-05-2020: created + +## Context + +Currently the RPC layer of Tendermint is using a variant of the JSON-RPC protocol. This ADR is meant to serve as a pro/con list for possible alternatives and JSON-RPC. + +There are currently two options being discussed: gRPC & JSON-RPC. + +### JSON-RPC + +JSON-RPC is a JSON-based RPC protocol. Tendermint has implemented its own variant of JSON-RPC which is not compatible with the [JSON-RPC 2.0 specification](https://www.jsonrpc.org/specification). + +**Pros:** + +- Easy to use & implement (by default) +- Well-known and well-understood by users and integrators +- Integrates reasonably well with web infrastructure (proxies, API gateways, service meshes, caches, etc) +- human readable encoding (by default) + +**Cons:** + +- No schema support +- RPC clients must be hand-written +- Streaming not built into protocol +- Underspecified types (e.g. numbers and timestamps) +- Tendermint has its own implementation (not standards compliant, maintenance overhead) + - High maintenance cost associated to this +- Stdlib `jsonrpc` package only supports JSON-RPC 1.0, no dominant package for JSON-RPC 2.0 +- Tooling around documentation/specification (e.g. Swagger) could be better +- JSON data is larger (offset by HTTP compression) +- Serializing is slow ([~100% marshal, ~400% unmarshal](https://github.com/alecthomas/go_serialization_benchmarks)); insignificant in absolute terms +- Specification was last updated in 2013 and is way behind Swagger/OpenAPI + +### gRPC + gRPC-gateway (REST + Swagger) + +gRPC is a high performant RPC framework. It has been battle tested by a large number of users and is heavily relied on and maintained by countless large corporations. + +**Pros:** + +- Efficient data retrieval for users, lite clients and other protocols +- Easily implemented in supported languages (Go, Dart, JS, TS, rust, Elixir, Haskell, ...) +- Defined schema with richer type system (Protocol Buffers) +- Can use common schemas and types across all protocols and data stores (RPC, ABCI, blocks, etc) +- Established conventions for forwards- and backwards-compatibility +- Bi-directional streaming +- Servers and clients are be autogenerated in many languages (e.g. Tendermint-rs) +- Auto-generated swagger documentation for REST API +- Backwards and forwards compatibility guarantees enforced at the protocol level. +- Can be used with different codecs (JSON, CBOR, ...) + +**Cons:** + +- Complex system involving cross-language schemas, code generation, and custom protocols +- Type system does not always map cleanly to native language type system; integration woes +- Many common types require Protobuf plugins (e.g. timestamps and duration) +- Generated code may be non-idiomatic and hard to use +- Migration will be disruptive and laborious + +## Decision + +> This section explains all of the details of the proposed solution, including implementation details. +> It should also describe affects / corollary items that may need to be changed as a part of this. +> If the proposed change will be large, please also indicate a way to do the change to maximize ease of review. +> (e.g. the optimal split of things to do between separate PR's) + +## Status + +> A decision may be "proposed" if it hasn't been agreed upon yet, or "accepted" once it is agreed upon. If a later ADR changes or reverses a decision, it may be marked as "deprecated" or "superseded" with a reference to its replacement. + +{Deprecated|Proposed|Accepted} + +## Consequences + +> This section describes the consequences, after applying the decision. All consequences should be summarized here, not just the "positive" ones. + +### Positive + +### Negative + +### Neutral + +## References + +> Are there any relevant PR comments, issues that led up to this, or articles referenced for why we made the given design choice? If so link them here! + +- {reference link} diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-058-event-hashing.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-058-event-hashing.md new file mode 100644 index 00000000..184b921d --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-058-event-hashing.md @@ -0,0 +1,122 @@ +# ADR 058: Event hashing + +## Changelog + +- 2020-07-17: initial version +- 2020-07-27: fixes after Ismail and Ethan's comments +- 2020-07-27: declined + +## Context + +Before [PR#4845](https://github.com/tendermint/tendermint/pull/4845), +`Header#LastResultsHash` was a root of the Merkle tree built from `DeliverTx` +results. Only `Code`, `Data` fields were included because `Info` and `Log` +fields are non-deterministic. + +At some point, we've added events to `ResponseBeginBlock`, `ResponseEndBlock`, +and `ResponseDeliverTx` to give applications a way to attach some additional +information to blocks / transactions. + +Many applications seem to have started using them since. + +However, before [PR#4845](https://github.com/tendermint/tendermint/pull/4845) +there was no way to prove that certain events were a part of the result +(_unless the application developer includes them into the state tree_). + +Hence, [PR#4845](https://github.com/tendermint/tendermint/pull/4845) was +opened. In it, `GasWanted` along with `GasUsed` are included when hashing +`DeliverTx` results. Also, events from `BeginBlock`, `EndBlock` and `DeliverTx` +results are hashed into the `LastResultsHash` as follows: + +- Since we do not expect `BeginBlock` and `EndBlock` to contain many events, + these will be Protobuf encoded and included in the Merkle tree as leaves. +- `LastResultsHash` therefore is the root hash of a Merkle tree w/ 3 leafs: + proto-encoded `ResponseBeginBlock#Events`, root hash of a Merkle tree build + from `ResponseDeliverTx` responses (Log, Info and Codespace fields are + ignored), and proto-encoded `ResponseEndBlock#Events`. +- Order of events is unchanged - same as received from the ABCI application. + +[Spec PR](https://github.com/tendermint/spec/pull/97/files) + +While it's certainly good to be able to prove something, introducing new events +or removing such becomes difficult because it breaks the `LastResultsHash`. It +means that every time you add, remove or update an event, you'll need a +hard-fork. And that is undoubtedly bad for applications, which are evolving and +don't have a stable events set. + +## Decision + +As a middle ground approach, the proposal is to add the +`Block#LastResultsEvents` consensus parameter that is a list of all events that +are to be hashed in the header. + +``` +@ proto/tendermint/abci/types.proto:295 @ message BlockParams { + int64 max_bytes = 1; + // Note: must be greater or equal to -1 + int64 max_gas = 2; + // List of events, which will be hashed into the LastResultsHash + repeated string last_results_events = 3; +} +``` + +Initially the list is empty. The ABCI application can change it via `InitChain` +or `EndBlock`. + +Example: + +```go +func (app *MyApp) DeliverTx(req types.RequestDeliverTx) types.ResponseDeliverTx { + //... + events := []abci.Event{ + { + Type: "transfer", + Attributes: []abci.EventAttribute{ + {Key: []byte("sender"), Value: []byte("Bob"), Index: true}, + }, + }, + } + return types.ResponseDeliverTx{Code: code.CodeTypeOK, Events: events} +} +``` + +For "transfer" event to be hashed, the `LastResultsEvents` must contain a +string "transfer". + +## Status + +Declined + +**Until there's more stability/motivation/use-cases/demand, the decision is to +push this entirely application side and just have apps which want events to be +provable to insert them into their application-side merkle trees. Of course +this puts more pressure on their application state and makes event proving +application specific, but it might help built up a better sense of use-cases +and how this ought to ultimately be done by Tendermint.** + +## Consequences + +### Positive + +1. networks can perform parameter change proposals to update this list as new events are added +2. allows networks to avoid having to do hard-forks +3. events can still be added at-will to the application w/o breaking anything + +### Negative + +1. yet another consensus parameter +2. more things to track in the tendermint state + +## References + +- [ADR 021](./adr-021-abci-events.md) +- [Indexing transactions](../app-dev/indexing-transactions.md) + +## Appendix A. Alternative proposals + +The other proposal was to add `Hash bool` flag to the `Event`, similarly to +`Index bool` EventAttribute's field. When `true`, Tendermint would hash it into +the `LastResultsEvents`. The downside is that the logic is implicit and depends +largely on the node's operator, who decides what application code to run. The +above proposal makes it (the logic) explicit and easy to upgrade via +governance. diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-059-evidence-composition-and-lifecycle.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-059-evidence-composition-and-lifecycle.md new file mode 100644 index 00000000..6e0f3f40 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-059-evidence-composition-and-lifecycle.md @@ -0,0 +1,306 @@ +# ADR 059: Evidence Composition and Lifecycle + +## Changelog + +- 04/09/2020: Initial Draft (Unabridged) +- 07/09/2020: First Version +- 13/03/2021: Ammendment to accomodate forward lunatic attack +- 29/06/2021: Add information about ABCI specific fields + +## Scope + +This document is designed to collate together and surface some predicaments involving evidence in Tendermint: both its composition and lifecycle. It then aims to find a solution to these. The scope does not extend to the verification nor detection of certain types of evidence but concerns itself mainly with the general form of evidence and how it moves from inception to application. + +## Background + +For a long time `DuplicateVoteEvidence`, formed in the consensus reactor, was the only evidence Tendermint had. It was produced whenever two votes from the same validator in the same round +was observed and thus it was designed that each evidence was for a single validator. It was predicted that there may come more forms of evidence and thus `DuplicateVoteEvidence` was used as the model for the `Evidence` interface and also for the form of the evidence data sent to the application. It is important to note that Tendermint concerns itself just with the detection and reporting of evidence and it is the responsibility of the application to exercise punishment. + +```go +type Evidence interface { //existing + Height() int64 // height of the offense + Time() time.Time // time of the offense + Address() []byte // address of the offending validator + Bytes() []byte // bytes which comprise the evidence + Hash() []byte // hash of the evidence + Verify(chainID string, pubKey crypto.PubKey) error // verify the evidence + Equal(Evidence) bool // check equality of evidence + + ValidateBasic() error + String() string +} +``` + +```go +type DuplicateVoteEvidence struct { + VoteA *Vote + VoteB *Vote + + timestamp time.Time // taken from the block time +} +``` + +Tendermint has now introduced a new type of evidence to protect light clients from being attacked. This `LightClientAttackEvidence` (see [here](https://github.com/informalsystems/tendermint-rs/blob/31ca3e64ce90786c1734caf186e30595832297a4/docs/spec/lightclient/attacks/evidence-handling.md) for more information) is vastly different to `DuplicateVoteEvidence` in that it is physically a much different size containing a complete signed header and validator set. It is formed within the light client, not the consensus reactor and requires a lot more information from state to verify (`VerifyLightClientAttack(commonHeader, trustedHeader *SignedHeader, commonVals *ValidatorSet)` vs `VerifyDuplicateVote(chainID string, pubKey PubKey)`). Finally it batches validators together (a single piece of evidence that implicates multiple malicious validators at a height) as opposed to having individual evidence (each piece of evidence is per validator per height). This evidence stretches the existing mould that was used to accommodate new types of evidence and has thus caused us to reconsider how evidence should be formatted and processed. + +```go +type LightClientAttackEvidence struct { // proposed struct in spec + ConflictingBlock *LightBlock + CommonHeight int64 + Type AttackType // enum: {Lunatic|Equivocation|Amnesia} + + timestamp time.Time // taken from the block time at the common height +} +``` +*Note: These three attack types have been proven by the research team to be exhaustive* + +## Possible Approaches for Evidence Composition + +### Individual framework + +Evidence remains on a per validator basis. This causes the least disruption to the current processes but requires that we break `LightClientAttackEvidence` into several pieces of evidence for each malicious validator. This not only has performance consequences in that there are n times as many database operations and that the gossiping of evidence will require more bandwidth then necessary (by requiring a header for each piece) but it potentially impacts our ability to validate it. In batch form, the full node can run the same process the light client did to see that 1/3 validating power was present in both the common block and the conflicting block whereas this becomes more difficult to verify individually without opening the possibility that malicious validators forge evidence against innocent . Not only that, but `LightClientAttackEvidence` also deals with amnesia attacks which unfortunately have the characteristic where we know the set of validators involved but not the subset that were actually malicious (more to be said about this later). And finally splitting the evidence into individual pieces makes it difficult to understand the severity of the attack (i.e. the total voting power involved in the attack) + +#### An example of a possible implementation path + +We would ignore amnesia evidence (as individually it's hard to make) and revert to the initial split we had before where `DuplicateVoteEvidence` is also used for light client equivocation attacks and thus we only need `LunaticEvidence`. We would also most likely need to remove `Verify` from the interface as this isn't really something that can be used. + +``` go +type LunaticEvidence struct { // individual lunatic attack + header *Header + commonHeight int64 + vote *Vote + + timestamp time.Time // once again taken from the block time at the height of the common header +} +``` + +### Batch Framework + +The last approach of this category would be to consider batch only evidence. This works fine with `LightClientAttackEvidence` but would require alterations to `DuplicateVoteEvidence` which would most likely mean that the consensus would send conflicting votes to a buffer in the evidence module which would then wrap all the votes together per height before gossiping them to other nodes and trying to commit it on chain. At a glance this may improve IO and verification speed and perhaps more importantly grouping validators gives the application and Tendermint a better overview of the severity of the attack. + +However individual evidence has the advantage that it is easy to check if a node already has that evidence meaning we just need to check hashes to know that we've already verified this evidence before. Batching evidence would imply that each node may have a different combination of duplicate votes which may complicate things. + +#### An example of a possible implementation path + +`LightClientAttackEvidence` won't change but the evidence interface will need to look like the proposed one above and `DuplicateVoteEvidence` will need to change to encompass multiple double votes. A problem with batch evidence is that it needs to be unique to avoid people from submitting different permutations. + +## Decision + +The decision is to adopt a hybrid design. + +We allow individual and batch evidence to coexist together, meaning that verification is done depending on the evidence type and that the bulk of the work is done in the evidence pool itself (including forming the evidence to be sent to the application). + + +## Detailed Design + +Evidence has the following simple interface: + +```go +type Evidence interface { //proposed + Height() int64 // height of the offense + Bytes() []byte // bytes which comprise the evidence + Hash() []byte // hash of the evidence + ValidateBasic() error + String() string +} +``` + +The changing of the interface is backwards compatible as these methods are all present in the previous version of the interface. However, networks will need to upgrade to be able to process the new evidence as verification has changed. + +We have two concrete types of evidence that fulfil this interface + +```go +type LightClientAttackEvidence struct { + ConflictingBlock *LightBlock + CommonHeight int64 // the last height at which the primary provider and witness provider had the same header + + // abci specific information + ByzantineValidators []*Validator // validators in the validator set that misbehaved in creating the conflicting block + TotalVotingPower int64 // total voting power of the validator set at the common height + Timestamp time.Time // timestamp of the block at the common height +} +``` +where the `Hash()` is the hash of the header and commonHeight. + +Note: It was also discussed whether to include the commit hash which captures the validators that signed the header. However this would open the opportunity for someone to propose multiple permutations of the same evidence (through different commit signatures) hence it was omitted. Consequentially, when it comes to verifying evidence in a block, for `LightClientAttackEvidence` we can't just check the hashes because someone could have the same hash as us but a different commit where less than 1/3 validators voted which would be an invalid version of the evidence. (see `fastCheck` for more details) + +```go +type DuplicateVoteEvidence { + VoteA *Vote + VoteB *Vote + + // abci specific information + TotalVotingPower int64 + ValidatorPower int64 + Timestamp time.Time +} +``` +where the `Hash()` is the hash of the two votes + +For both of these types of evidence, `Bytes()` represents the proto-encoded byte array format of the evidence and `ValidateBasic` is +an initial consistency check to make sure the evidence has a valid structure. + +### The Evidence Pool + +`LightClientAttackEvidence` is generated in the light client and `DuplicateVoteEvidence` in consensus. Both are sent to the evidence pool through `AddEvidence(ev Evidence) error`. The evidence pool's primary purpose is to verify evidence. It also gossips evidence to other peers' evidence pool and serves it to consensus so it can be committed on chain and the relevant information can be sent to the application in order to exercise punishment. When evidence is added, the pool first runs `Has(ev Evidence)` to check if it has already received it (by comparing hashes) and then `Verify(ev Evidence) error`. Once verified the evidence pool stores it it's pending database. There are two databases: one for pending evidence that is not yet committed and another of the committed evidence (to avoid committing evidence twice) + +#### Verification + +`Verify()` does the following: + +- Use the hash to see if we already have this evidence in our committed database. + +- Use the height to check if the evidence hasn't expired. + +- If it has expired then use the height to find the block header and check if the time has also expired in which case we drop the evidence + +- Then proceed with switch statement for each of the two evidence: + +For `DuplicateVote`: + +- Check that height, round, type and validator address are the same + +- Check that the Block ID is different + +- Check the look up table for addresses to make sure there already isn't evidence against this validator + +- Fetch the validator set and confirm that the address is in the set at the height of the attack + +- Check that the chain ID and signature is valid. + +For `LightClientAttack` + +- Fetch the common signed header and val set from the common height and use skipping verification to verify the conflicting header + +- Fetch the trusted signed header at the same height as the conflicting header and compare with the conflicting header to work out which type of attack it is and in doing so return the malicious validators. NOTE: If the node doesn't have the signed header at the height of the conflicting header, it instead fetches the latest header it has and checks to see if it can prove the evidence based on a violation of header time. This is known as forward lunatic attack. + + - If equivocation, return the validators that signed for the commits of both the trusted and signed header + + - If lunatic, return the validators from the common val set that signed in the conflicting block + + - If amnesia, return no validators (since we can't know which validators are malicious). This also means that we don't currently send amnesia evidence to the application, although we will introduce more robust amnesia evidence handling in future Tendermint Core releases + +- Check that the hashes of the conflicting header and the trusted header are different + +- In the case of a forward lunatic attack, where the trusted header height is less than the conflicting header height, the node checks that the time of the trusted header is later than the time of conflicting header. This proves that the conflicting header breaks monotonically increasing time. If the node doesn't have a trusted header with a later time then it is unable to validate the evidence for now. + +- Lastly, for each validator, check the look up table to make sure there already isn't evidence against this validator + +After verification we persist the evidence with the key `height/hash` to the pending evidence database in the evidence pool. + +#### ABCI Evidence + +Both evidence structures contain data (such as timestamp) that are necessary to be passed to the application but do not strictly constitute evidence of misbehavior. As such, these fields are verified last. If any of these fields are invalid to a node i.e. they don't correspond with their state, nodes will reconstruct a new evidence struct from the existing fields and repopulate the abci specific fields with their own state data. + +#### Broadcasting and receiving evidence + +The evidence pool also runs a reactor that broadcasts the newly validated +evidence to all connected peers. + +Receiving evidence from other evidence reactors works in the same manner as receiving evidence from the consensus reactor or a light client. + + +#### Proposing evidence on the block + +When it comes to prevoting and precomitting a proposal that contains evidence, the full node will once again +call upon the evidence pool to verify the evidence using `CheckEvidence(ev []Evidence)`: + +This performs the following actions: + +1. Loops through all the evidence to check that nothing has been duplicated + +2. For each evidence, run `fastCheck(ev evidence)` which works similar to `Has` but instead for `LightClientAttackEvidence` if it has the +same hash it then goes on to check that the validators it has are all signers in the commit of the conflicting header. If it doesn't pass fast check (because it hasn't seen the evidence before) then it will have to verify the evidence. + +3. runs `Verify(ev Evidence)` - Note: this also saves the evidence to the db as mentioned before. + + +#### Updating application and pool + +The final part of the lifecycle is when the block is committed and the `BlockExecutor` then updates state. As part of this process, the `BlockExecutor` gets the evidence pool to create a simplified format for the evidence to be sent to the application. This happens in `ApplyBlock` where the executor calls `Update(Block, State) []abci.Evidence`. + +```go +abciResponses.BeginBlock.ByzantineValidators = evpool.Update(block, state) +``` + +Here is the format of the evidence that the application will receive. As seen above, this is stored as an array within `BeginBlock`. +The changes to the application are minimal (it is still formed one for each malicious validator) with the exception of using an enum instead of a string for the evidence type. + +```go +type Evidence struct { + // either LightClientAttackEvidence or DuplicateVoteEvidence as an enum (abci.EvidenceType) + Type EvidenceType `protobuf:"varint,1,opt,name=type,proto3,enum=tendermint.abci.EvidenceType" json:"type,omitempty"` + // The offending validator + Validator Validator `protobuf:"bytes,2,opt,name=validator,proto3" json:"validator"` + // The height when the offense occurred + Height int64 `protobuf:"varint,3,opt,name=height,proto3" json:"height,omitempty"` + // The corresponding time where the offense occurred + Time time.Time `protobuf:"bytes,4,opt,name=time,proto3,stdtime" json:"time"` + // Total voting power of the validator set in case the ABCI application does + // not store historical validators. + // https://github.com/tendermint/tendermint/issues/4581 + TotalVotingPower int64 `protobuf:"varint,5,opt,name=total_voting_power,json=totalVotingPower,proto3" json:"total_voting_power,omitempty"` +} +``` + + +This `Update()` function does the following: + +- Increments state which keeps track of both the current time and height used for measuring expiry + +- Marks evidence as committed and saves to db. This prevents validators from proposing committed evidence in the future + Note: the db just saves the height and the hash. There is no need to save the entire committed evidence + +- Forms ABCI evidence as such: (note for `DuplicateVoteEvidence` the validators array size is 1) + ```go + for _, val := range evInfo.Validators { + abciEv = append(abciEv, &abci.Evidence{ + Type: evType, // either DuplicateVote or LightClientAttack + Validator: val, // the offending validator (which includes the address, pubkey and power) + Height: evInfo.ev.Height(), // the height when the offense happened + Time: evInfo.time, // the time when the offense happened + TotalVotingPower: evInfo.totalVotingPower // the total voting power of the validator set + }) + } + ``` + +- Removes expired evidence from both pending and committed databases + +The ABCI evidence is then sent via the `BlockExecutor` to the application. + +#### Summary + +To summarize, we can see the lifecycle of evidence as such: + +![evidence_lifecycle](./img/evidence_lifecycle.png) + +Evidence is first detected and created in the light client and consensus reactor. It is verified and stored as `EvidenceInfo` and gossiped to the evidence pools in other nodes. The consensus reactor later communicates with the evidence pool to either retrieve evidence to be put into a block, or verify the evidence the consensus reactor has retrieved in a block. Lastly when a block is added to the chain, the block executor sends the committed evidence back to the evidence pool so a pointer to the evidence can be stored in the evidence pool and it can update it's height and time. Finally, it turns the committed evidence into ABCI evidence and through the block executor passes the evidence to the application so the application can handle it. + +## Status + +Implemented + +## Consequences + + + +### Positive + +- Evidence is better contained to the evidence pool / module +- LightClientAttack is kept together (easier for verification and bandwidth) +- Variations on commit sigs in LightClientAttack doesn't lead to multiple permutations and multiple evidence +- Address to evidence map prevents DOS attacks, where a single validator could DOS the network by flooding it with evidence submissions + +### Negative + +- Changes the `Evidence` interface and thus is a block breaking change +- Changes the ABCI `Evidence` and is thus a ABCI breaking change +- Unable to query evidence for address / time without evidence pool + +### Neutral + + +## References + + + +- [LightClientAttackEvidence](https://github.com/informalsystems/tendermint-rs/blob/31ca3e64ce90786c1734caf186e30595832297a4/docs/spec/lightclient/attacks/evidence-handling.md) diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-060-go-api-stability.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-060-go-api-stability.md new file mode 100644 index 00000000..31b70e73 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-060-go-api-stability.md @@ -0,0 +1,193 @@ +# ADR 060: Go API Stability + +## Changelog + +- 2020-09-08: Initial version. (@erikgrinaker) + +- 2020-09-09: Tweak accepted changes, add initial public API packages, add consequences. (@erikgrinaker) + +- 2020-09-17: Clarify initial public API. (@erikgrinaker) + +## Context + +With the release of Tendermint 1.0 we will adopt [semantic versioning](https://semver.org). One major implication is a guarantee that we will not make backwards-incompatible changes until Tendermint 2.0 (except in pre-release versions). In order to provide this guarantee for our Go API, we must clearly define which of our APIs are public, and what changes are considered backwards-compatible. + +Currently, we list packages that we consider public in our [README](https://github.com/tendermint/tendermint#versioning), but since we are still at version 0.x we do not provide any backwards compatiblity guarantees at all. + +### Glossary + +* **External project:** a different Git/VCS repository or code base. + +* **External package:** a different Go package, can be a child or sibling package in the same project. + +* **Internal code:** code not intended for use in external projects. + +* **Internal directory:** code under `internal/` which cannot be imported in external projects. + +* **Exported:** a Go identifier starting with an uppercase letter, which can therefore be accessed by an external package. + +* **Private:** a Go identifier starting with a lowercase letter, which therefore cannot be accessed by an external package unless via an exported field, variable, or function/method return value. + +* **Public API:** any Go identifier that can be imported or accessed by an external project, except test code in `_test.go` files. + +* **Private API:** any Go identifier that is not accessible via a public API, including all code in the internal directory. + +## Alternative Approaches + +- Split all public APIs out to separate Go modules in separate Git repositories, and consider all Tendermint code internal and not subject to API backwards compatibility at all. This was rejected, since it has been attempted by the Tendermint project earlier, resulting in too much dependency management overhead. + +- Simply document which APIs are public and which are private. This is the current approach, but users should not be expected to self-enforce this, the documentation is not always up-to-date, and external projects will often end up depending on internal code anyway. + +## Decision + +From Tendermint 1.0, all internal code (except private APIs) will be placed in a root-level [`internal` directory](https://golang.org/cmd/go/#hdr-Internal_Directories), which the Go compiler will block for use by external projects. All exported items outside of the `internal` directory are considered a public API and subject to backwards compatibility guarantees, except files ending in `_test.go`. + +The `crypto` package may be split out to a separate module in a separate repo. This is the main general-purpose package used by external projects, and is the only Tendermint dependency in e.g. IAVL which can cause some problems for projects depending on both IAVL and Tendermint. This will be decided after further discussion. + +The `tm-db` package will remain a separate module in a separate repo. The `crypto` package may possibly be split out, pending further discussion, as this is the main general-purpose package used by other projects. + +## Detailed Design + +### Public API + +When preparing our public API for 1.0, we should keep these principles in mind: + +- Limit the number of public APIs that we start out with - we can always add new APIs later, but we can't change or remove APIs once they're made public. + +- Before an API is made public, do a thorough review of the API to make sure it covers any future needs, can accomodate expected changes, and follows good API design practices. + +The following is the minimum set of public APIs that will be included in 1.0, in some form: + +- `abci` +- packages used for constructing nodes `config`, `libs/log`, and `version` +- Client APIs, i.e. `rpc/client`, `light`, and `privval`. +- `crypto` (possibly as a separate repo) + +We may offer additional APIs as well, following further discussions internally and with other stakeholders. However, public APIs for providing custom components (e.g. reactors and mempools) are not planned for 1.0, but may be added in a later 1.x version if this is something we want to offer. + +For comparison, the following are the number of Tendermint imports in the Cosmos SDK (excluding tests), which should be mostly satisfied by the planned APIs. + +``` + 1 github.com/tendermint/tendermint/abci/server + 73 github.com/tendermint/tendermint/abci/types + 2 github.com/tendermint/tendermint/cmd/tendermint/commands + 7 github.com/tendermint/tendermint/config + 68 github.com/tendermint/tendermint/crypto + 1 github.com/tendermint/tendermint/crypto/armor + 10 github.com/tendermint/tendermint/crypto/ed25519 + 2 github.com/tendermint/tendermint/crypto/encoding + 3 github.com/tendermint/tendermint/crypto/merkle + 3 github.com/tendermint/tendermint/crypto/sr25519 + 8 github.com/tendermint/tendermint/crypto/tmhash + 1 github.com/tendermint/tendermint/crypto/xsalsa20symmetric + 11 github.com/tendermint/tendermint/libs/bytes + 2 github.com/tendermint/tendermint/libs/bytes.HexBytes + 15 github.com/tendermint/tendermint/libs/cli + 2 github.com/tendermint/tendermint/libs/cli/flags + 2 github.com/tendermint/tendermint/libs/json + 30 github.com/tendermint/tendermint/libs/log + 1 github.com/tendermint/tendermint/libs/math + 11 github.com/tendermint/tendermint/libs/os + 4 github.com/tendermint/tendermint/libs/rand + 1 github.com/tendermint/tendermint/libs/strings + 5 github.com/tendermint/tendermint/light + 1 github.com/tendermint/tendermint/internal/mempool + 3 github.com/tendermint/tendermint/node + 5 github.com/tendermint/tendermint/internal/p2p + 4 github.com/tendermint/tendermint/privval + 10 github.com/tendermint/tendermint/proto/tendermint/crypto + 1 github.com/tendermint/tendermint/proto/tendermint/libs/bits + 24 github.com/tendermint/tendermint/proto/tendermint/types + 3 github.com/tendermint/tendermint/proto/tendermint/version + 2 github.com/tendermint/tendermint/proxy + 3 github.com/tendermint/tendermint/rpc/client + 1 github.com/tendermint/tendermint/rpc/client/http + 2 github.com/tendermint/tendermint/rpc/client/local + 3 github.com/tendermint/tendermint/rpc/core/types + 1 github.com/tendermint/tendermint/rpc/jsonrpc/server + 33 github.com/tendermint/tendermint/types + 2 github.com/tendermint/tendermint/types/time + 1 github.com/tendermint/tendermint/version +``` + +### Backwards-Compatible Changes + +In Go, [almost all API changes are backwards-incompatible](https://blog.golang.org/module-compatibility) and thus exported items in public APIs generally cannot be changed until Tendermint 2.0. The only backwards-compatible changes we can make to public APIs are: + +- Adding a package. + +- Adding a new identifier to the package scope (e.g. const, var, func, struct, interface, etc.). + +- Adding a new method to a struct. + +- Adding a new field to a struct, if the zero-value preserves any old behavior. + +- Changing the order of fields in a struct. + +- Adding a variadic parameter to a named function or struct method, if the function type itself is not assignable in any public APIs (e.g. a callback). + +- Adding a new method to an interface, or a variadic parameter to an interface method, _if the interface already has a private method_ (which prevents external packages from implementing it). + +- Widening a numeric type as long as it is a named type (e.g. `type Number int32` can change to `int64`, but not `int8` or `uint32`). + +Note that public APIs can expose private types (e.g. via an exported variable, field, or function/method return value), in which case the exported fields and methods on these private types are also part of the public API and covered by its backwards compatiblity guarantees. In general, private types should never be accessible via public APIs unless wrapped in an exported interface. + +Also note that if we accept, return, export, or embed types from a dependency, we assume the backwards compatibility responsibility for that dependency, and must make sure any dependency upgrades comply with the above constraints. + +We should run CI linters for minor version branches to enforce this, e.g. [apidiff](https://go.googlesource.com/exp/+/refs/heads/master/apidiff/README.md), [breakcheck](https://github.com/gbbr/breakcheck), and [apicombat](https://github.com/bradleyfalzon/apicompat). + +#### Accepted Breakage + +The above changes can still break programs in a few ways - these are _not_ considered backwards-incompatible changes, and users are advised to avoid this usage: + +- If a program uses unkeyed struct literals (e.g. `Foo{"bar", "baz"}`) and we add fields or change the field order, the program will no longer compile or may have logic errors. + +- If a program embeds two structs in a struct, and we add a new field or method to an embedded Tendermint struct which also exists in the other embedded struct, the program will no longer compile. + +- If a program compares two structs (e.g. with `==`), and we add a new field of an incomparable type (slice, map, func, or struct that contains these) to a Tendermint struct which is compared, the program will no longer compile. + +- If a program assigns a Tendermint function to an identifier, and we add a variadic parameter to the function signature, the program will no longer compile. + +### Strategies for API Evolution + +The API guarantees above can be fairly constraining, but are unavoidable given the Go language design. The following tricks can be employed where appropriate to allow us to make changes to the API: + +- We can add a new function or method with a different name that takes additional parameters, and have the old function call the new one. + +- Functions and methods can take an options struct instead of separate parameters, to allow adding new options - this is particularly suitable for functions that take many parameters and are expected to be extended, and especially for interfaces where we cannot add new methods with different parameters at all. + +- Interfaces can include a private method, e.g. `interface { private() }`, to make them unimplementable by external packages and thus allow us to add new methods to the interface without breaking other programs. Of course, this can't be used for interfaces that should be implementable externally. + +- We can use [interface upgrades](https://avtok.com/2014/11/05/interface-upgrades.html) to allow implementers of an existing interface to also implement a new interface, as long as the old interface can still be used - e.g. the new interface `BetterReader` may have a method `ReadBetter()`, and a function that takes a `Reader` interface as an input can check if the implementer also implements `BetterReader` and in that case call `ReadBetter()` instead of `Read()`. + +## Status + +Proposed + +## Consequences + +### Positive + +- Users can safely upgrade with less fear of applications breaking, and know whether an upgrade only includes bug fixes or also functional enhancements + +- External developers have a predictable and well-defined API to build on that will be supported for some time + +- Less synchronization between teams, since there is a clearer contract and timeline for changes and they happen less frequently + +- More documentation will remain accurate, since it's not chasing a moving target + +- Less time will be spent on code churn and more time spent on functional improvements, both for the community and for our teams + +### Negative + +- Many improvements, changes, and bug fixes will have to be postponed until the next major version, possibly for a year or more + +- The pace of development will slow down, since we must work within the existing API constraints, and spend more time planning public APIs + +- External developers may lose access to some currently exported APIs and functionality + +## References + +- [#4451: Place internal APIs under internal package](https://github.com/tendermint/tendermint/issues/4451) + +- [On Pluggability](https://docs.google.com/document/d/1G08LnwSyb6BAuCVSMF3EKn47CGdhZ5wPZYJQr4-bw58/edit?ts=5f609f11) diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-061-p2p-refactor-scope.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-061-p2p-refactor-scope.md new file mode 100644 index 00000000..eedff207 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-061-p2p-refactor-scope.md @@ -0,0 +1,109 @@ +# ADR 061: P2P Refactor Scope + +## Changelog + +- 2020-10-30: Initial version (@erikgrinaker) + +## Context + +The `p2p` package responsible for peer-to-peer networking is rather old and has a number of weaknesses, including tight coupling, leaky abstractions, lack of tests, DoS vulnerabilites, poor performance, custom protocols, and incorrect behavior. A refactor has been discussed for several years ([#2067](https://github.com/tendermint/tendermint/issues/2067)). + +Informal Systems are also building a Rust implementation of Tendermint, [Tendermint-rs](https://github.com/informalsystems/tendermint-rs), and plan to implement P2P networking support over the next year. As part of this work, they have requested adopting e.g. [QUIC](https://datatracker.ietf.org/doc/draft-ietf-quic-transport/) as a transport protocol instead of implementing the custom application-level `MConnection` stream multiplexing protocol that Tendermint currently uses. + +This ADR summarizes recent discussion with stakeholders on the scope of a P2P refactor. Specific designs and implementations will be submitted as separate ADRs. + +## Alternative Approaches + +There have been recurring proposals to adopt [LibP2P](https://libp2p.io) instead of maintaining our own P2P networking stack (see [#3696](https://github.com/tendermint/tendermint/issues/3696)). While this appears to be a good idea in principle, it would be a highly breaking protocol change, there are indications that we might have to fork and modify LibP2P, and there are concerns about the abstractions used. + +In discussions with Informal Systems we decided to begin with incremental improvements to the current P2P stack, add support for pluggable transports, and then gradually start experimenting with LibP2P as a transport layer. If this proves successful, we can consider adopting it for higher-level components at a later time. + +## Decision + +The P2P stack will be refactored and improved iteratively, in several phases: + +* **Phase 1:** code and API refactoring, maintaining protocol compatibility as far as possible. + +* **Phase 2:** additional transports and incremental protocol improvements. + +* **Phase 3:** disruptive protocol changes. + +The scope of phases 2 and 3 is still uncertain, and will be revisited once the preceding phases have been completed as we'll have a better sense of requirements and challenges. + +## Detailed Design + +Separate ADRs will be submitted for specific designs and changes in each phase, following research and prototyping. Below are objectives in order of priority. + +### Phase 1: Code and API Refactoring + +This phase will focus on improving the internal abstractions and implementations in the `p2p` package. As far as possible, it should not change the P2P protocol in a backwards-incompatible way. + +* Cleaner, decoupled abstractions for e.g. `Reactor`, `Switch`, and `Peer`. [#2067](https://github.com/tendermint/tendermint/issues/2067) [#5287](https://github.com/tendermint/tendermint/issues/5287) [#3833](https://github.com/tendermint/tendermint/issues/3833) + * Reactors should receive messages in separate goroutines or via buffered channels. [#2888](https://github.com/tendermint/tendermint/issues/2888) +* Improved peer lifecycle management. [#3679](https://github.com/tendermint/tendermint/issues/3679) [#3719](https://github.com/tendermint/tendermint/issues/3719) [#3653](https://github.com/tendermint/tendermint/issues/3653) [#3540](https://github.com/tendermint/tendermint/issues/3540) [#3183](https://github.com/tendermint/tendermint/issues/3183) [#3081](https://github.com/tendermint/tendermint/issues/3081) [#1356](https://github.com/tendermint/tendermint/issues/1356) + * Peer prioritization. [#2860](https://github.com/tendermint/tendermint/issues/2860) [#2041](https://github.com/tendermint/tendermint/issues/2041) +* Pluggable transports, with `MConnection` as one implementation. [#5587](https://github.com/tendermint/tendermint/issues/5587) [#2430](https://github.com/tendermint/tendermint/issues/2430) [#805](https://github.com/tendermint/tendermint/issues/805) +* Improved peer address handling. + * Address book refactor. [#4848](https://github.com/tendermint/tendermint/issues/4848) [#2661](https://github.com/tendermint/tendermint/issues/2661) + * Transport-agnostic peer addressing. [#5587](https://github.com/tendermint/tendermint/issues/5587) [#3782](https://github.com/tendermint/tendermint/issues/3782) [#3692](https://github.com/tendermint/tendermint/issues/3692) + * Improved detection and advertisement of own address. [#5588](https://github.com/tendermint/tendermint/issues/5588) [#4260](https://github.com/tendermint/tendermint/issues/4260) [#3716](https://github.com/tendermint/tendermint/issues/3716) [#1727](https://github.com/tendermint/tendermint/issues/1727) + * Support multiple IPs per peer. [#1521](https://github.com/tendermint/tendermint/issues/1521) [#2317](https://github.com/tendermint/tendermint/issues/2317) + +The refactor should attempt to address the following secondary objectives: testability, observability, performance, security, quality-of-service, backpressure, and DoS resilience. Much of this will be revisited as explicit objectives in phase 2. + +Ideally, the refactor should happen incrementally, with regular merges to `master` every few weeks. This will take more time overall, and cause frequent breaking changes to internal Go APIs, but it reduces the branch drift and gets the code tested sooner and more broadly. + +### Phase 2: Additional Transports and Protocol Improvements + +This phase will focus on protocol improvements and other breaking changes. The following are considered proposals that will need to be evaluated separately once the refactor is done. Additional proposals are likely to be added during phase 1. + +* QUIC transport. [#198](https://github.com/tendermint/spec/issues/198) +* Noise protocol for secret connection handshake. [#5589](https://github.com/tendermint/tendermint/issues/5589) [#3340](https://github.com/tendermint/tendermint/issues/3340) +* Peer ID in connection handshake. [#5590](https://github.com/tendermint/tendermint/issues/5590) +* Peer and service discovery (e.g. RPC nodes, state sync snapshots). [#5481](https://github.com/tendermint/tendermint/issues/5481) [#4583](https://github.com/tendermint/tendermint/issues/4583) +* Rate-limiting, backpressure, and QoS scheduling. [#4753](https://github.com/tendermint/tendermint/issues/4753) [#2338](https://github.com/tendermint/tendermint/issues/2338) +* Compression. [#2375](https://github.com/tendermint/tendermint/issues/2375) +* Improved metrics and tracing. [#3849](https://github.com/tendermint/tendermint/issues/3849) [#2600](https://github.com/tendermint/tendermint/issues/2600) +* Simplified P2P configuration options. + +### Phase 3: Disruptive Protocol Changes + +This phase covers speculative, wide-reaching proposals that are poorly defined and highly uncertain. They will be evaluated once the previous phases are done. + +* Adopt LibP2P. [#3696](https://github.com/tendermint/tendermint/issues/3696) +* Allow cross-reactor communication, possibly without channels. +* Dynamic channel advertisment, as reactors are enabled/disabled. [#4394](https://github.com/tendermint/tendermint/issues/4394) [#1148](https://github.com/tendermint/tendermint/issues/1148) +* Pubsub-style networking topology and pattern. +* Support multiple chain IDs in the same network. + +## Status + +Proposed + +## Consequences + +### Positive + +* Cleaner, simpler architecture that's easier to reason about and test, and thus hopefully less buggy. + +* Improved performance and robustness. + +* Reduced maintenance burden and increased interoperability by the possible adoption of standardized protocols such as QUIC and Noise. + +* Improved usability, with better observability, simpler configuration, and more automation (e.g. peer/service/address discovery, rate-limiting, and backpressure). + +### Negative + +* Maintaining our own P2P networking stack is resource-intensive. + +* Abstracting away the underlying transport may prevent usage of advanced transport features. + +* Breaking changes to APIs and protocols are disruptive to users. + +## References + +See issue links above. + +- [#2067: P2P Refactor](https://github.com/tendermint/tendermint/issues/2067) + +- [P2P refactor brainstorm document](https://docs.google.com/document/d/1FUTADZyLnwA9z7ndayuhAdAFRKujhh_y73D0ZFdKiOQ/edit?pli=1#) diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-062-p2p-architecture.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-062-p2p-architecture.md new file mode 100644 index 00000000..b2456da4 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-062-p2p-architecture.md @@ -0,0 +1,617 @@ +# ADR 062: P2P Architecture and Abstractions + +## Changelog + +- 2020-11-09: Initial version (@erikgrinaker) + +- 2020-11-13: Remove stream IDs, move peer errors onto channel, note on moving PEX into core (@erikgrinaker) + +- 2020-11-16: Notes on recommended reactor implementation patterns, approve ADR (@erikgrinaker) + +- 2021-02-04: Update with new P2P core and Transport API changes (@erikgrinaker). + +## Context + +In [ADR 061](adr-061-p2p-refactor-scope.md) we decided to refactor the peer-to-peer (P2P) networking stack. The first phase is to redesign and refactor the internal P2P architecture, while retaining protocol compatibility as far as possible. + +## Alternative Approaches + +Several variations of the proposed design were considered, including e.g. calling interface methods instead of passing messages (like the current architecture), merging channels with streams, exposing the internal peer data structure to reactors, being message format-agnostic via arbitrary codecs, and so on. This design was chosen because it has very loose coupling, is simpler to reason about and more convenient to use, avoids race conditions and lock contention for internal data structures, gives reactors better control of message ordering and processing semantics, and allows for QoS scheduling and backpressure in a very natural way. + +[multiaddr](https://github.com/multiformats/multiaddr) was considered as a transport-agnostic peer address format over regular URLs, but it does not appear to have very widespread adoption, and advanced features like protocol encapsulation and tunneling do not appear to be immediately useful to us. + +There were also proposals to use LibP2P instead of maintaining our own P2P stack, which were rejected (for now) in [ADR 061](adr-061-p2p-refactor-scope.md). + +The initial version of this ADR had a byte-oriented multi-stream transport API, but this had to be abandoned/postponed to maintain backwards-compatibility with the existing MConnection protocol which is message-oriented. See the rejected RFC in [tendermint/spec#227](https://github.com/tendermint/spec/pull/227) for details. + +## Decision + +The P2P stack will be redesigned as a message-oriented architecture, primarily relying on Go channels for communication and scheduling. It will use a message-oriented transport to binary messages with individual peers, bidirectional peer-addressable channels to send and receive Protobuf messages, a router to route messages between reactors and peers, and a peer manager to manage peer lifecycle information. Message passing is asynchronous with at-most-once delivery. + +## Detailed Design + +This ADR is primarily concerned with the architecture and interfaces of the P2P stack, not implementation details. The interfaces described here should therefore be considered a rough architecture outline, not a complete and final design. + +Primary design objectives have been: + +* Loose coupling between components, for a simpler, more robust, and test-friendly architecture. +* Pluggable transports (not necessarily networked). +* Better scheduling of messages, with improved prioritization, backpressure, and performance. +* Centralized peer lifecycle and connection management. +* Better peer address detection, advertisement, and exchange. +* Wire-level backwards compatibility with current P2P network protocols, except where it proves too obstructive. + +The main abstractions in the new stack are: + +* `Transport`: An arbitrary mechanism to exchange binary messages with a peer across a `Connection`. +* `Channel`: A bidirectional channel to asynchronously exchange Protobuf messages with peers using node ID addressing. +* `Router`: Maintains transport connections to relevant peers and routes channel messages. +* `PeerManager`: Manages peer lifecycle information, e.g. deciding which peers to dial and when, using a `peerStore` for storage. +* Reactor: A design pattern loosely defined as "something which listens on a channel and reacts to messages". + +These abstractions are illustrated in the following diagram (representing the internals of node A) and described in detail below. + +![P2P Architecture Diagram](img/adr-062-architecture.svg) + +### Transports + +Transports are arbitrary mechanisms for exchanging binary messages with a peer. For example, a gRPC transport would connect to a peer over TCP/IP and send data using the gRPC protocol, while an in-memory transport might communicate with a peer running in another goroutine using internal Go channels. Note that transports don't have a notion of a "peer" or "node" as such - instead, they establish connections between arbitrary endpoint addresses (e.g. IP address and port number), to decouple them from the rest of the P2P stack. + +Transports must satisfy the following requirements: + +* Be connection-oriented, and support both listening for inbound connections and making outbound connections using endpoint addresses. + +* Support sending binary messages with distinct channel IDs (although channels and channel IDs are a higher-level application protocol concept explained in the Router section, they are threaded through the transport layer as well for backwards compatibilty with the existing MConnection protocol). + +* Exchange the MConnection `NodeInfo` and public key via a node handshake, and possibly encrypt or sign the traffic as appropriate. + +The initial transport is a port of the current MConnection protocol currently used by Tendermint, and should be backwards-compatible at the wire level. An in-memory transport for testing has also been implemented. There are plans to explore a QUIC transport that may replace the MConnection protocol. + +The `Transport` interface is as follows: + +```go +// Transport is a connection-oriented mechanism for exchanging data with a peer. +type Transport interface { + // Protocols returns the protocols supported by the transport. The Router + // uses this to pick a transport for an Endpoint. + Protocols() []Protocol + + // Endpoints returns the local endpoints the transport is listening on, if any. + // How to listen is transport-dependent, e.g. MConnTransport uses Listen() while + // MemoryTransport starts listening via MemoryNetwork.CreateTransport(). + Endpoints() []Endpoint + + // Accept waits for the next inbound connection on a listening endpoint, blocking + // until either a connection is available or the transport is closed. On closure, + // io.EOF is returned and further Accept calls are futile. + Accept() (Connection, error) + + // Dial creates an outbound connection to an endpoint. + Dial(context.Context, Endpoint) (Connection, error) + + // Close stops accepting new connections, but does not close active connections. + Close() error +} +``` + +How the transport configures listening is transport-dependent, and not covered by the interface. This typically happens during transport construction, where a single instance of the transport is created and set to listen on an appropriate network interface before being passed to the router. + +#### Endpoints + +`Endpoint` represents a transport endpoint (e.g. an IP address and port). A connection always has two endpoints: one at the local node and one at the remote peer. Outbound connections to remote endpoints are made via `Dial()`, and inbound connections to listening endpoints are returned via `Accept()`. + +The `Endpoint` struct is: + +```go +// Endpoint represents a transport connection endpoint, either local or remote. +// +// Endpoints are not necessarily networked (see e.g. MemoryTransport) but all +// networked endpoints must use IP as the underlying transport protocol to allow +// e.g. IP address filtering. Either IP or Path (or both) must be set. +type Endpoint struct { + // Protocol specifies the transport protocol. + Protocol Protocol + + // IP is an IP address (v4 or v6) to connect to. If set, this defines the + // endpoint as a networked endpoint. + IP net.IP + + // Port is a network port (either TCP or UDP). If 0, a default port may be + // used depending on the protocol. + Port uint16 + + // Path is an optional transport-specific path or identifier. + Path string +} + +// Protocol identifies a transport protocol. +type Protocol string +``` + +Endpoints are arbitrary transport-specific addresses, but if they are networked they must use IP addresses and thus rely on IP as a fundamental packet routing protocol. This enables policies for address discovery, advertisement, and exchange - for example, a private `192.168.0.0/24` IP address should only be advertised to peers on that IP network, while the public address `8.8.8.8` may be advertised to all peers. Similarly, any port numbers if given must represent TCP and/or UDP port numbers, in order to use [UPnP](https://en.wikipedia.org/wiki/Universal_Plug_and_Play) to autoconfigure e.g. NAT gateways. + +Non-networked endpoints (without an IP address) are considered local, and will only be advertised to other peers connecting via the same protocol. For example, the in-memory transport used for testing uses `Endpoint{Protocol: "memory", Path: "foo"}` as an address for the node "foo", and this should only be advertised to other nodes using `Protocol: "memory"`. + +#### Connections + +A connection represents an established transport connection between two endpoints (i.e. two nodes), which can be used to exchange binary messages with logical channel IDs (corresponding to the higher-level channel IDs used in the router). Connections are set up either via `Transport.Dial()` (outbound) or `Transport.Accept()` (inbound). + +Once a connection is esablished, `Transport.Handshake()` must be called to perform a node handshake, exchanging node info and public keys to verify node identities. Node handshakes should not really be part of the transport layer (it's an application protocol concern), this exists for backwards-compatibility with the existing MConnection protocol which conflates the two. `NodeInfo` is part of the existing MConnection protocol, but does not appear to be documented in the specification -- refer to the Go codebase for details. + +The `Connection` interface is shown below. It omits certain additions that are currently implemented for backwards compatibility with the legacy P2P stack and are planned to be removed before the final release. + +```go +// Connection represents an established connection between two endpoints. +type Connection interface { + // Handshake executes a node handshake with the remote peer. It must be + // called once the connection is established, and returns the remote peer's + // node info and public key. The caller is responsible for validation. + Handshake(context.Context, NodeInfo, crypto.PrivKey) (NodeInfo, crypto.PubKey, error) + + // ReceiveMessage returns the next message received on the connection, + // blocking until one is available. Returns io.EOF if closed. + ReceiveMessage() (ChannelID, []byte, error) + + // SendMessage sends a message on the connection. Returns io.EOF if closed. + SendMessage(ChannelID, []byte) error + + // LocalEndpoint returns the local endpoint for the connection. + LocalEndpoint() Endpoint + + // RemoteEndpoint returns the remote endpoint for the connection. + RemoteEndpoint() Endpoint + + // Close closes the connection. + Close() error +} +``` + +This ADR initially proposed a byte-oriented multi-stream connection API that follows more typical networking API conventions (using e.g. `io.Reader` and `io.Writer` interfaces which easily compose with other libraries). This would also allow moving the responsibility for message framing, node handshakes, and traffic scheduling to the common router instead of reimplementing this across transports, and would allow making better use of multi-stream protocols such as QUIC. However, this would require minor breaking changes to the MConnection protocol which were rejected, see [tendermint/spec#227](https://github.com/tendermint/spec/pull/227) for details. This should be revisited when starting work on a QUIC transport. + +### Peer Management + +Peers are other Tendermint nodes. Each peer is identified by a unique `NodeID` (tied to the node's private key). + +#### Peer Addresses + +Nodes have one or more `NodeAddress` addresses expressed as URLs that they can be reached at. Examples of node addresses might be e.g.: + +* `mconn://nodeid@host.domain.com:25567/path` +* `memory:nodeid` + +Addresses are resolved into one or more transport endpoints, e.g. by resolving DNS hostnames into IP addresses. Peers should always be expressed as address URLs rather than endpoints (which are a lower-level transport construct). + +```go +// NodeID is a hex-encoded crypto.Address. It must be lowercased +// (for uniqueness) and of length 40. +type NodeID string + +// NodeAddress is a node address URL. It differs from a transport Endpoint in +// that it contains the node's ID, and that the address hostname may be resolved +// into multiple IP addresses (and thus multiple endpoints). +// +// If the URL is opaque, i.e. of the form "scheme:opaque", then the opaque part +// is expected to contain a node ID. +type NodeAddress struct { + NodeID NodeID + Protocol Protocol + Hostname string + Port uint16 + Path string +} + +// ParseNodeAddress parses a node address URL into a NodeAddress, normalizing +// and validating it. +func ParseNodeAddress(urlString string) (NodeAddress, error) + +// Resolve resolves a NodeAddress into a set of Endpoints, e.g. by expanding +// out a DNS hostname to IP addresses. +func (a NodeAddress) Resolve(ctx context.Context) ([]Endpoint, error) +``` + +#### Peer Manager + +The P2P stack needs to track a lot of internal state about peers, such as their addresses, connection state, priorities, availability, failures, retries, and so on. This responsibility has been separated out to a `PeerManager`, which track this state for the `Router` (but does not maintain the actual transport connections themselves, which is the router's responsibility). + +The `PeerManager` is a synchronous state machine, where all state transitions are serialized (implemented as synchronous method calls holding an exclusive mutex lock). Most peer state is intentionally kept internal, stored in a `peerStore` database that persists it as appropriate, and the external interfaces pass the minimum amount of information necessary in order to avoid shared state between router goroutines. This design significantly simplifies the model, making it much easier to reason about and test than if it was baked into the asynchronous ball of concurrency that the P2P networking core must necessarily be. As peer lifecycle events are expected to be relatively infrequent, this should not significantly impact performance either. + +The `Router` uses the `PeerManager` to request which peers to dial and evict, and reports in with peer lifecycle events such as connections, disconnections, and failures as they occur. The manager can reject these events (e.g. reject an inbound connection) by returning errors. This happens as follows: + +* Outbound connections, via `Transport.Dial`: + * `DialNext()`: returns a peer address to dial, or blocks until one is available. + * `DialFailed()`: reports a peer dial failure. + * `Dialed()`: reports a peer dial success. + * `Ready()`: reports the peer as routed and ready. + * `Disconnected()`: reports a peer disconnection. + +* Inbound connections, via `Transport.Accept`: + * `Accepted()`: reports an inbound peer connection. + * `Ready()`: reports the peer as routed and ready. + * `Disconnected()`: reports a peer disconnection. + +* Evictions, via `Connection.Close`: + * `EvictNext()`: returns a peer to disconnect, or blocks until one is available. + * `Disconnected()`: reports a peer disconnection. + +These calls have the following interface: + +```go +// DialNext returns a peer address to dial, blocking until one is available. +func (m *PeerManager) DialNext(ctx context.Context) (NodeAddress, error) + +// DialFailed reports a dial failure for the given address. +func (m *PeerManager) DialFailed(address NodeAddress) error + +// Dialed reports a successful outbound connection to the given address. +func (m *PeerManager) Dialed(address NodeAddress) error + +// Accepted reports a successful inbound connection from the given node. +func (m *PeerManager) Accepted(peerID NodeID) error + +// Ready reports the peer as fully routed and ready for use. +func (m *PeerManager) Ready(peerID NodeID) error + +// EvictNext returns a peer ID to disconnect, blocking until one is available. +func (m *PeerManager) EvictNext(ctx context.Context) (NodeID, error) + +// Disconnected reports a peer disconnection. +func (m *PeerManager) Disconnected(peerID NodeID) error +``` + +Internally, the `PeerManager` uses a numeric peer score to prioritize peers, e.g. when deciding which peers to dial next. The scoring policy has not yet been implemented, but should take into account e.g. node configuration such a `persistent_peers`, uptime and connection failures, performance, and so on. The manager will also attempt to automatically upgrade to better-scored peers by evicting lower-scored peers when a better one becomes available (e.g. when a persistent peer comes back online after an outage). + +The `PeerManager` should also have an API for reporting peer behavior from reactors that affects its score (e.g. signing a block increases the score, double-voting decreases it or even bans the peer), but this has not yet been designed and implemented. + +Additionally, the `PeerManager` provides `PeerUpdates` subscriptions that will receive `PeerUpdate` events whenever significant peer state changes happen. Reactors can use these e.g. to know when peers are connected or disconnected, and take appropriate action. This is currently fairly minimal: + +```go +// Subscribe subscribes to peer updates. The caller must consume the peer updates +// in a timely fashion and close the subscription when done, to avoid stalling the +// PeerManager as delivery is semi-synchronous, guaranteed, and ordered. +func (m *PeerManager) Subscribe() *PeerUpdates + +// PeerUpdate is a peer update event sent via PeerUpdates. +type PeerUpdate struct { + NodeID NodeID + Status PeerStatus +} + +// PeerStatus is a peer status. +type PeerStatus string + +const ( + PeerStatusUp PeerStatus = "up" // Connected and ready. + PeerStatusDown PeerStatus = "down" // Disconnected. +) + +// PeerUpdates is a real-time peer update subscription. +type PeerUpdates struct { ... } + +// Updates returns a channel for consuming peer updates. +func (pu *PeerUpdates) Updates() <-chan PeerUpdate + +// Close closes the peer updates subscription. +func (pu *PeerUpdates) Close() +``` + +The `PeerManager` will also be responsible for providing peer information to the PEX reactor that can be gossipped to other nodes. This requires an improved system for peer address detection and advertisement, that e.g. reliably detects peer and self addresses and only gossips private network addresses to other peers on the same network, but this system has not yet been fully designed and implemented. + +### Channels + +While low-level data exchange happens via the `Transport`, the high-level API is based on a bidirectional `Channel` that can send and receive Protobuf messages addressed by `NodeID`. A channel is identified by an arbitrary `ChannelID` identifier, and can exchange Protobuf messages of one specific type (since the type to unmarshal into must be predefined). Message delivery is asynchronous and at-most-once. + +The channel can also be used to report peer errors, e.g. when receiving an invalid or malignant message. This may cause the peer to be disconnected or banned depending on `PeerManager` policy, but should probably be replaced by a broader peer behavior API that can also report good behavior. + +A `Channel` has this interface: + +```go +// ChannelID is an arbitrary channel ID. +type ChannelID uint16 + +// Channel is a bidirectional channel to exchange Protobuf messages with peers. +type Channel struct { + ID ChannelID // Channel ID. + In <-chan Envelope // Inbound messages (peers to reactors). + Out chan<- Envelope // outbound messages (reactors to peers) + Error chan<- PeerError // Peer error reporting. + messageType proto.Message // Channel's message type, for e.g. unmarshaling. +} + +// Close closes the channel, also closing Out and Error. +func (c *Channel) Close() error + +// Envelope specifies the message receiver and sender. +type Envelope struct { + From NodeID // Sender (empty if outbound). + To NodeID // Receiver (empty if inbound). + Broadcast bool // Send to all connected peers, ignoring To. + Message proto.Message // Message payload. +} + +// PeerError is a peer error reported via the Error channel. +type PeerError struct { + NodeID NodeID + Err error +} +``` + +A channel can reach any connected peer, and will automatically (un)marshal the Protobuf messages. Message scheduling and queueing is a `Router` implementation concern, and can use any number of algorithms such as FIFO, round-robin, priority queues, etc. Since message delivery is not guaranteed, both inbound and outbound messages may be dropped, buffered, reordered, or blocked as appropriate. + +Since a channel can only exchange messages of a single type, it is often useful to use a wrapper message type with e.g. a Protobuf `oneof` field that specifies a set of inner message types that it can contain. The channel can automatically perform this (un)wrapping if the outer message type implements the `Wrapper` interface (see [Reactor Example](#reactor-example) for an example): + +```go +// Wrapper is a Protobuf message that can contain a variety of inner messages. +// If a Channel's message type implements Wrapper, the channel will +// automatically (un)wrap passed messages using the container type, such that +// the channel can transparently support multiple message types. +type Wrapper interface { + proto.Message + + // Wrap will take a message and wrap it in this one. + Wrap(proto.Message) error + + // Unwrap will unwrap the inner message contained in this message. + Unwrap() (proto.Message, error) +} +``` + +### Routers + +The router exeutes P2P networking for a node, taking instructions from and reporting events to the `PeerManager`, maintaining transport connections to peers, and routing messages between channels and peers. + +Practically all concurrency in the P2P stack has been moved into the router and reactors, while as many other responsibilities as possible have been moved into separate components such as the `Transport` and `PeerManager` that can remain largely synchronous. Limiting concurrency to a single core component makes it much easier to reason about since there is only a single concurrency structure, while the remaining components can be serial, simple, and easily testable. + +The `Router` has a very minimal API, since it is mostly driven by `PeerManager` and `Transport` events: + +```go +// Router maintains peer transport connections and routes messages between +// peers and channels. +type Router struct { + // Some details have been omitted below. + + logger log.Logger + options RouterOptions + nodeInfo NodeInfo + privKey crypto.PrivKey + peerManager *PeerManager + transports []Transport + + peerMtx sync.RWMutex + peerQueues map[NodeID]queue + + channelMtx sync.RWMutex + channelQueues map[ChannelID]queue +} + +// OpenChannel opens a new channel for the given message type. The caller must +// close the channel when done, before stopping the Router. messageType is the +// type of message passed through the channel. +func (r *Router) OpenChannel(id ChannelID, messageType proto.Message) (*Channel, error) + +// Start starts the router, connecting to peers and routing messages. +func (r *Router) Start() error + +// Stop stops the router, disconnecting from all peers and stopping message routing. +func (r *Router) Stop() error +``` + +All Go channel sends in the `Router` and reactors are blocking (the router also selects on signal channels for closure and shutdown). The responsibility for message scheduling, prioritization, backpressure, and load shedding is centralized in a core `queue` interface that is used at contention points (i.e. from all peers to a single channel, and from all channels to a single peer): + +```go +// queue does QoS scheduling for Envelopes, enqueueing and dequeueing according +// to some policy. Queues are used at contention points, i.e.: +// - Receiving inbound messages to a single channel from all peers. +// - Sending outbound messages to a single peer from all channels. +type queue interface { + // enqueue returns a channel for submitting envelopes. + enqueue() chan<- Envelope + + // dequeue returns a channel ordered according to some queueing policy. + dequeue() <-chan Envelope + + // close closes the queue. After this call enqueue() will block, so the + // caller must select on closed() as well to avoid blocking forever. The + // enqueue() and dequeue() channels will not be closed. + close() + + // closed returns a channel that's closed when the scheduler is closed. + closed() <-chan struct{} +} +``` + +The current implementation is `fifoQueue`, which is a simple unbuffered lossless queue that passes messages in the order they were received and blocks until the message is delivered (i.e. it is a Go channel). The router will need a more sophisticated queueing policy, but this has not yet been implemented. + +The internal `Router` goroutine structure and design is described in the `Router` GoDoc, which is included below for reference: + +```go +// On startup, three main goroutines are spawned to maintain peer connections: +// +// dialPeers(): in a loop, calls PeerManager.DialNext() to get the next peer +// address to dial and spawns a goroutine that dials the peer, handshakes +// with it, and begins to route messages if successful. +// +// acceptPeers(): in a loop, waits for an inbound connection via +// Transport.Accept() and spawns a goroutine that handshakes with it and +// begins to route messages if successful. +// +// evictPeers(): in a loop, calls PeerManager.EvictNext() to get the next +// peer to evict, and disconnects it by closing its message queue. +// +// When a peer is connected, an outbound peer message queue is registered in +// peerQueues, and routePeer() is called to spawn off two additional goroutines: +// +// sendPeer(): waits for an outbound message from the peerQueues queue, +// marshals it, and passes it to the peer transport which delivers it. +// +// receivePeer(): waits for an inbound message from the peer transport, +// unmarshals it, and passes it to the appropriate inbound channel queue +// in channelQueues. +// +// When a reactor opens a channel via OpenChannel, an inbound channel message +// queue is registered in channelQueues, and a channel goroutine is spawned: +// +// routeChannel(): waits for an outbound message from the channel, looks +// up the recipient peer's outbound message queue in peerQueues, and submits +// the message to it. +// +// All channel sends in the router are blocking. It is the responsibility of the +// queue interface in peerQueues and channelQueues to prioritize and drop +// messages as appropriate during contention to prevent stalls and ensure good +// quality of service. +``` + +### Reactor Example + +While reactors are a first-class concept in the current P2P stack (i.e. there is an explicit `p2p.Reactor` interface), they will simply be a design pattern in the new stack, loosely defined as "something which listens on a channel and reacts to messages". + +Since reactors have very few formal constraints, they can be implemented in a variety of ways. There is currently no recommended pattern for implementing reactors, to avoid overspecification and scope creep in this ADR. However, prototyping and developing a reactor pattern should be done early during implementation, to make sure reactors built using the `Channel` interface can satisfy the needs for convenience, deterministic tests, and reliability. + +Below is a trivial example of a simple echo reactor implemented as a function. The reactor will exchange the following Protobuf messages: + +```protobuf +message EchoMessage { + oneof inner { + PingMessage ping = 1; + PongMessage pong = 2; + } +} + +message PingMessage { + string content = 1; +} + +message PongMessage { + string content = 1; +} +``` + +Implementing the `Wrapper` interface for `EchoMessage` allows transparently passing `PingMessage` and `PongMessage` through the channel, where it will automatically be (un)wrapped in an `EchoMessage`: + +```go +func (m *EchoMessage) Wrap(inner proto.Message) error { + switch inner := inner.(type) { + case *PingMessage: + m.Inner = &EchoMessage_PingMessage{Ping: inner} + case *PongMessage: + m.Inner = &EchoMessage_PongMessage{Pong: inner} + default: + return fmt.Errorf("unknown message %T", inner) + } + return nil +} + +func (m *EchoMessage) Unwrap() (proto.Message, error) { + switch inner := m.Inner.(type) { + case *EchoMessage_PingMessage: + return inner.Ping, nil + case *EchoMessage_PongMessage: + return inner.Pong, nil + default: + return nil, fmt.Errorf("unknown message %T", inner) + } +} +``` + +The reactor itself would be implemented e.g. like this: + +```go +// RunEchoReactor wires up an echo reactor to a router and runs it. +func RunEchoReactor(router *p2p.Router, peerManager *p2p.PeerManager) error { + channel, err := router.OpenChannel(1, &EchoMessage{}) + if err != nil { + return err + } + defer channel.Close() + peerUpdates := peerManager.Subscribe() + defer peerUpdates.Close() + + return EchoReactor(context.Background(), channel, peerUpdates) +} + +// EchoReactor provides an echo service, pinging all known peers until the given +// context is canceled. +func EchoReactor(ctx context.Context, channel *p2p.Channel, peerUpdates *p2p.PeerUpdates) error { + ticker := time.NewTicker(5 * time.Second) + defer ticker.Stop() + + for { + select { + // Send ping message to all known peers every 5 seconds. + case <-ticker.C: + channel.Out <- Envelope{ + Broadcast: true, + Message: &PingMessage{Content: "👋"}, + } + + // When we receive a message from a peer, either respond to ping, output + // pong, or report peer error on unknown message type. + case envelope := <-channel.In: + switch msg := envelope.Message.(type) { + case *PingMessage: + channel.Out <- Envelope{ + To: envelope.From, + Message: &PongMessage{Content: msg.Content}, + } + + case *PongMessage: + fmt.Printf("%q replied with %q\n", envelope.From, msg.Content) + + default: + channel.Error <- PeerError{ + PeerID: envelope.From, + Err: fmt.Errorf("unexpected message %T", msg), + } + } + + // Output info about any peer status changes. + case peerUpdate := <-peerUpdates: + fmt.Printf("Peer %q changed status to %q", peerUpdate.PeerID, peerUpdate.Status) + + // Exit when context is canceled. + case <-ctx.Done(): + return nil + } + } +} +``` + +## Status + +Proposed + +Was partially implemented in v0.35 ([#5670](https://github.com/tendermint/tendermint/issues/5670)) + +## Consequences + +### Positive + +* Reduced coupling and simplified interfaces should lead to better understandability, increased reliability, and more testing. + +* Using message passing via Go channels gives better control of backpressure and quality-of-service scheduling. + +* Peer lifecycle and connection management is centralized in a single entity, making it easier to reason about. + +* Detection, advertisement, and exchange of node addresses will be improved. + +* Additional transports (e.g. QUIC) can be implemented and used in parallel with the existing MConn protocol. + +* The P2P protocol will not be broken in the initial version, if possible. + +### Negative + +* Fully implementing the new design as indended is likely to require breaking changes to the P2P protocol at some point, although the initial implementation shouldn't. + +* Gradually migrating the existing stack and maintaining backwards-compatibility will be more labor-intensive than simply replacing the entire stack. + +* A complete overhaul of P2P internals is likely to cause temporary performance regressions and bugs as the implementation matures. + +* Hiding peer management information inside the `PeerManager` may prevent certain functionality or require additional deliberate interfaces for information exchange, as a tradeoff to simplify the design, reduce coupling, and avoid race conditions and lock contention. + +### Neutral + +* Implementation details around e.g. peer management, message scheduling, and peer and endpoint advertisement are not yet determined. + +## References + +* [ADR 061: P2P Refactor Scope](adr-061-p2p-refactor-scope.md) +* [#5670 p2p: internal refactor and architecture redesign](https://github.com/tendermint/tendermint/issues/5670) diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-063-privval-grpc.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-063-privval-grpc.md new file mode 100644 index 00000000..d3b6cc84 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-063-privval-grpc.md @@ -0,0 +1,109 @@ +# ADR 063: Privval gRPC + +## Changelog + +- 23/11/2020: Initial Version (@marbar3778) + +## Context + +Validators use remote signers to help secure their keys. This system is Tendermint's recommended way to secure validators, but the path to integration with Tendermint's private validator client is plagued with custom protocols. + +Tendermint uses its own custom secure connection protocol (`SecretConnection`) and a raw tcp/unix socket connection protocol. The secure connection protocol until recently was exposed to man in the middle attacks and can take longer to integrate if not using Golang. The raw tcp connection protocol is less custom, but has been causing minute issues with users. + +Migrating Tendermint's private validator client to a widely adopted protocol, gRPC, will ease the current maintenance and integration burden experienced with the current protocol. + +## Decision + +After discussing with multiple stake holders, [gRPC](https://grpc.io/) was decided on to replace the current private validator protocol. gRPC is a widely adopted protocol in the micro-service and cloud infrastructure world. gRPC uses [protocol-buffers](https://developers.google.com/protocol-buffers) to describe its services, providing a language agnostic implementation. Tendermint uses protobuf for on disk and over the wire encoding already making the integration with gRPC simpler. + +## Alternative Approaches + +- JSON-RPC: We did not consider JSON-RPC because Tendermint uses protobuf extensively making gRPC a natural choice. + +## Detailed Design + +With the recent integration of [Protobuf](https://developers.google.com/protocol-buffers) into Tendermint the needed changes to migrate from the current private validator protocol to gRPC is not large. + +The [service definition](https://grpc.io/docs/what-is-grpc/core-concepts/#service-definition) for gRPC will be defined as: + +```proto + service PrivValidatorAPI { + rpc GetPubKey(tendermint.proto.privval.PubKeyRequest) returns (tendermint.proto.privval.PubKeyResponse); + rpc SignVote(tendermint.proto.privval.SignVoteRequest) returns (tendermint.proto.privval.SignedVoteResponse); + rpc SignProposal(tendermint.proto.privval.SignProposalRequest) returns (tendermint.proto.privval.SignedProposalResponse); + + message PubKeyRequest { + string chain_id = 1; + } + + // PubKeyResponse is a response message containing the public key. + message PubKeyResponse { + tendermint.crypto.PublicKey pub_key = 1 [(gogoproto.nullable) = false]; + } + + // SignVoteRequest is a request to sign a vote + message SignVoteRequest { + tendermint.types.Vote vote = 1; + string chain_id = 2; + } + + // SignedVoteResponse is a response containing a signed vote or an error + message SignedVoteResponse { + tendermint.types.Vote vote = 1 [(gogoproto.nullable) = false]; + } + + // SignProposalRequest is a request to sign a proposal + message SignProposalRequest { + tendermint.types.Proposal proposal = 1; + string chain_id = 2; + } + + // SignedProposalResponse is response containing a signed proposal or an error + message SignedProposalResponse { + tendermint.types.Proposal proposal = 1 [(gogoproto.nullable) = false]; + } +} +``` + +> Note: Remote Singer errors are removed in favor of [grpc status error codes](https://grpc.io/docs/guides/error/). + +In previous versions of the remote signer, Tendermint acted as the server and the remote signer as the client. In this process the client established a long lived connection providing a way for the server to make requests to the client. In the new version it has been simplified. Tendermint is the client and the remote signer is the server. This follows client and server architecture and simplifies the previous protocol. + +#### Keep Alive + +If you have worked on the private validator system you will see that we are removing the `PingRequest` and `PingResponse` messages. These messages were used to create functionality which kept the connection alive. With gRPC there is a [keep alive feature](https://github.com/grpc/grpc/blob/master/doc/keepalive.md) that will be added along side the integration to provide the same functionality. + +#### Metrics + +Remote signers are crucial to operating secure and consistently up Validators. In the past there were no metrics to tell the operator if something is wrong other than the node not signing. Integrating metrics into the client and provided server will be done with [prometheus](https://github.com/grpc-ecosystem/go-grpc-prometheus). This will be integrated into node's prometheus export for node operators. + +#### Security + +[TLS](https://en.wikipedia.org/wiki/Transport_Layer_Security) is widely adopted with the use of gRPC. There are various forms of TLS (one-way & two-way). One way is the client identifying who the server is, while two way is both parties identifying the other. For Tendermint's use case having both parties identifying each other provides adds an extra layer of security. This requires users to generate both client and server certificates for a TLS connection. + +An insecure option will be provided for users who do not wish to secure the connection. + +#### Upgrade Path + +This is a largely breaking change for validator operators. The optimal upgrade path would be to release gRPC in a minor release, allow key management systems to migrate to the new protocol. In the next major release the current system (raw tcp/unix) is removed. This allows users to migrate to the new system and not have to coordinate upgrading the key management system alongside a network upgrade. + +The upgrade of [tmkms](https://github.com/iqlusioninc/tmkms) will be coordinated with Iqlusion. They will be able to make the necessary upgrades to allow users to migrate to gRPC from the current protocol. + +## Status + +Accepted (tracked in +[\#9256](https://github.com/tendermint/tendermint/issues/9256)) + +### Positive + +- Use an adopted standard for secure communication. (TLS) +- Use an adopted communication protocol. (gRPC) +- Requests are multiplexed onto the tcp connection. (http/2) +- Language agnostic service definition. + +### Negative + +- Users will need to generate certificates to use TLS. (Added step) +- Users will need to find a supported gRPC supported key management system + +### Neutral diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-064-batch-verification.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-064-batch-verification.md new file mode 100644 index 00000000..13bba25e --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-064-batch-verification.md @@ -0,0 +1,90 @@ +# ADR 064: Batch Verification + +## Changelog + +- January 28, 2021: Created (@marbar3778) + +## Context + +Tendermint uses public private key cryptography for validator signing. When a block is proposed and voted on validators sign a message representing acceptance of a block, rejection is signaled via a nil vote. These signatures are also used to verify previous blocks are correct if a node is syncing. Currently, Tendermint requires each signature to be verified individually, this leads to a slow down of block times. + +Batch Verification is the process of taking many messages, keys, and signatures adding them together and verifying them all at once. The public key can be the same in which case it would mean a single user is signing many messages. In our case each public key is unique, each validator has their own and contribute a unique message. The algorithm can vary from curve to curve but the performance benefit, over single verifying messages, public keys and signatures is shared. + +## Alternative Approaches + +- Signature aggregation + - Signature aggregation is an alternative to batch verification. Signature aggregation leads to fast verification and smaller block sizes. At the time of writing this ADR there is on going work to enable signature aggregation in Tendermint. The reason why we have opted to not introduce it at this time is because every validator signs a unique message. + Signing a unique message prevents aggregation before verification. For example if we were to implement signature aggregation with BLS, there could be a potential slow down of 10x-100x in verification speeds. + +## Decision + +Adopt Batch Verification. + +## Detailed Design + +A new interface will be introduced. This interface will have three methods `NewBatchVerifier`, `Add` and `VerifyBatch`. + +```go +type BatchVerifier interface { + Add(key crypto.Pubkey, signature, message []byte) error // Add appends an entry into the BatchVerifier. + Verify() bool // Verify verifies all the entries in the BatchVerifier. If the verification fails it is unknown which entry failed and each entry will need to be verified individually. +} +``` + +- `NewBatchVerifier` creates a new verifier. This verifier will be populated with entries to be verified. +- `Add` adds an entry to the Verifier. Add accepts a public key and two slice of bytes (signature and message). +- `Verify` verifies all the entires. At the end of Verify if the underlying API does not reset the Verifier to its initial state (empty), it should be done here. This prevents accidentally reusing the verifier with entries from a previous verification. + +Above there is mention of an entry. An entry can be constructed in many ways depending on the needs of the underlying curve. A simple approach would be: + +```go +type entry struct { + pubKey crypto.Pubkey + signature []byte + message []byte +} +``` + +The main reason this approach is being taken is to prevent simple mistakes. Some APIs allow the user to create three slices and pass them to the `VerifyBatch` function but this relies on the user to safely generate all the slices (see example below). We would like to minimize the possibility of making a mistake. + +```go +func Verify(keys []crypto.Pubkey, signatures, messages[][]byte) bool +``` + +This change will not affect any users in anyway other than faster verification times. + +This new api will be used for verification in both consensus and block syncing. Within the current Verify functions there will be a check to see if the key types supports the BatchVerification API. If it does it will execute batch verification, if not single signature verification will be used. + +#### Consensus + + The process within consensus will be to wait for 2/3+ of the votes to be received, once they are received `Verify()` will be called to batch verify all the messages. The messages that come in after 2/3+ has been verified will be individually verified. + +#### Block Sync & Light Client + + The process for block sync & light client verification will be to verify only 2/3+ in a batch style. Since these processes are not participating in consensus there is no need to wait for more messages. + +If batch verifications fails for any reason, it will not be known which entry caused the failure. Verification will need to revert to single signature verification. + +Starting out, only ed25519 will support batch verification. + +## Status + +Implemented + +### Positive + +- Faster verification times, if the curve supports it + +### Negative + +- No way to see which key failed verification + - A failure means reverting back to single signature verification. + +### Neutral + +## References + +[Ed25519 Library](https://github.com/hdevalence/ed25519consensus) +[Ed25519 spec](https://ed25519.cr.yp.to/) +[Signature Aggregation for votes](https://github.com/tendermint/tendermint/issues/1319) +[Proposer-based timestamps](https://github.com/tendermint/tendermint/issues/2840) diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-065-custom-event-indexing.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-065-custom-event-indexing.md new file mode 100644 index 00000000..5679bc48 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-065-custom-event-indexing.md @@ -0,0 +1,425 @@ +# ADR 065: Custom Event Indexing + +- [ADR 065: Custom Event Indexing](#adr-065-custom-event-indexing) + - [Changelog](#changelog) + - [Status](#status) + - [Context](#context) + - [Alternative Approaches](#alternative-approaches) + - [Decision](#decision) + - [Detailed Design](#detailed-design) + - [EventSink](#eventsink) + - [Supported Sinks](#supported-sinks) + - [`KVEventSink`](#kveventsink) + - [`PSQLEventSink`](#psqleventsink) + - [Configuration](#configuration) + - [Future Improvements](#future-improvements) + - [Consequences](#consequences) + - [Positive](#positive) + - [Negative](#negative) + - [Neutral](#neutral) + - [References](#references) + +## Changelog + +- April 1, 2021: Initial Draft (@alexanderbez) +- April 28, 2021: Specify search capabilities are only supported through the KV indexer (@marbar3778) +- May 19, 2021: Update the SQL schema and the eventsink interface (@jayt106) +- Aug 30, 2021: Update the SQL schema and the psql implementation (@creachadair) +- Oct 5, 2021: Clarify goals and implementation changes (@creachadair) + +## Status + +Implemented + +## Context + +Currently, Tendermint Core supports block and transaction event indexing through +the `tx_index.indexer` configuration. Events are captured in transactions and +are indexed via a `TxIndexer` type. Events are captured in blocks, specifically +from `BeginBlock` and `EndBlock` application responses, and are indexed via a +`BlockIndexer` type. Both of these types are managed by a single `IndexerService` +which is responsible for consuming events and sending those events off to be +indexed by the respective type. + +In addition to indexing, Tendermint Core also supports the ability to query for +both indexed transaction and block events via Tendermint's RPC layer. The ability +to query for these indexed events facilitates a great multitude of upstream client +and application capabilities, e.g. block explorers, IBC relayers, and auxiliary +data availability and indexing services. + +Currently, Tendermint only supports indexing via a `kv` indexer, which is supported +by an underlying embedded key/value store database. The `kv` indexer implements +its own indexing and query mechanisms. While the former is somewhat trivial, +providing a rich and flexible query layer is not as trivial and has caused many +issues and UX concerns for upstream clients and applications. + +The fragile nature of the proprietary `kv` query engine and the potential +performance and scaling issues that arise when a large number of consumers are +introduced, motivate the need for a more robust and flexible indexing and query +solution. + +## Alternative Approaches + +With regards to alternative approaches to a more robust solution, the only serious +contender that was considered was to transition to using [SQLite](https://www.sqlite.org/index.html). + +While the approach would work, it locks us into a specific query language and +storage layer, so in some ways it's only a bit better than our current approach. +In addition, the implementation would require the introduction of CGO into the +Tendermint Core stack, whereas right now CGO is only introduced depending on +the database used. + +## Decision + +We will adopt a similar approach to that of the Cosmos SDK's `KVStore` state +listening described in [ADR-038](https://github.com/cosmos/cosmos-sdk/blob/master/docs/architecture/adr-038-state-listening.md). + +We will implement the following changes: + +- Introduce a new interface, `EventSink`, that all data sinks must implement. +- Augment the existing `tx_index.indexer` configuration to now accept a series + of one or more indexer types, i.e., sinks. +- Combine the current `TxIndexer` and `BlockIndexer` into a single `KVEventSink` + that implements the `EventSink` interface. +- Introduce an additional `EventSink` implementation that is backed by + [PostgreSQL](https://www.postgresql.org/). + - Implement the necessary schemas to support both block and transaction event indexing. +- Update `IndexerService` to use a series of `EventSinks`. + +In addition: + +- The Postgres indexer implementation will _not_ implement the proprietary `kv` + query language. Users wishing to write queries against the Postgres indexer + will connect to the underlying DBMS directly and use SQL queries based on the + indexing schema. + + Future custom indexer implementations will not be required to support the + proprietary query language either. + +- For now, the existing `kv` indexer will be left in place with its current + query support, but will be marked as deprecated in a subsequent release, and + the documentation will be updated to encourage users who need to query the + event index to migrate to the Postgres indexer. + +- In the future we may remove the `kv` indexer entirely, or replace it with a + different implementation; that decision is deferred as future work. + +- In the future, we may remove the index query endpoints from the RPC service + entirely; that decision is deferred as future work, but recommended. + + +## Detailed Design + +### EventSink + +We introduce the `EventSink` interface type that all supported sinks must implement. +The interface is defined as follows: + +```go +type EventSink interface { + IndexBlockEvents(types.EventDataNewBlockHeader) error + IndexTxEvents([]*abci.TxResult) error + + SearchBlockEvents(context.Context, *query.Query) ([]int64, error) + SearchTxEvents(context.Context, *query.Query) ([]*abci.TxResult, error) + + GetTxByHash([]byte) (*abci.TxResult, error) + HasBlock(int64) (bool, error) + + Type() EventSinkType + Stop() error +} +``` + +The `IndexerService` will accept a list of one or more `EventSink` types. During +the `OnStart` method it will call the appropriate APIs on each `EventSink` to +index both block and transaction events. + +### Supported Sinks + +We will initially support two `EventSink` types out of the box. + +#### `KVEventSink` + +This type of `EventSink` is a combination of the `TxIndexer` and `BlockIndexer` +indexers, both of which are backed by a single embedded key/value database. + +A bulk of the existing business logic will remain the same, but the existing APIs +mapped to the new `EventSink` API. Both types will be removed in favor of a single +`KVEventSink` type. + +The `KVEventSink` will be the only `EventSink` enabled by default, so from a UX +perspective, operators should not notice a difference apart from a configuration +change. + +We omit `EventSink` implementation details as it should be fairly straightforward +to map the existing business logic to the new APIs. + +#### `PSQLEventSink` + +This type of `EventSink` indexes block and transaction events into a [PostgreSQL](https://www.postgresql.org/). +database. We define and automatically migrate the following schema when the +`IndexerService` starts. + +The postgres eventsink will not support `tx_search`, `block_search`, `GetTxByHash` and `HasBlock`. + +```sql +-- Table Definition ---------------------------------------------- + +-- The blocks table records metadata about each block. +-- The block record does not include its events or transactions (see tx_results). +CREATE TABLE blocks ( + rowid BIGSERIAL PRIMARY KEY, + + height BIGINT NOT NULL, + chain_id VARCHAR NOT NULL, + + -- When this block header was logged into the sink, in UTC. + created_at TIMESTAMPTZ NOT NULL, + + UNIQUE (height, chain_id) +); + +-- Index blocks by height and chain, since we need to resolve block IDs when +-- indexing transaction records and transaction events. +CREATE INDEX idx_blocks_height_chain ON blocks(height, chain_id); + +-- The tx_results table records metadata about transaction results. Note that +-- the events from a transaction are stored separately. +CREATE TABLE tx_results ( + rowid BIGSERIAL PRIMARY KEY, + + -- The block to which this transaction belongs. + block_id BIGINT NOT NULL REFERENCES blocks(rowid), + -- The sequential index of the transaction within the block. + index INTEGER NOT NULL, + -- When this result record was logged into the sink, in UTC. + created_at TIMESTAMPTZ NOT NULL, + -- The hex-encoded hash of the transaction. + tx_hash VARCHAR NOT NULL, + -- The protobuf wire encoding of the TxResult message. + tx_result BYTEA NOT NULL, + + UNIQUE (block_id, index) +); + +-- The events table records events. All events (both block and transaction) are +-- associated with a block ID; transaction events also have a transaction ID. +CREATE TABLE events ( + rowid BIGSERIAL PRIMARY KEY, + + -- The block and transaction this event belongs to. + -- If tx_id is NULL, this is a block event. + block_id BIGINT NOT NULL REFERENCES blocks(rowid), + tx_id BIGINT NULL REFERENCES tx_results(rowid), + + -- The application-defined type label for the event. + type VARCHAR NOT NULL +); + +-- The attributes table records event attributes. +CREATE TABLE attributes ( + event_id BIGINT NOT NULL REFERENCES events(rowid), + key VARCHAR NOT NULL, -- bare key + composite_key VARCHAR NOT NULL, -- composed type.key + value VARCHAR NULL, + + UNIQUE (event_id, key) +); + +-- A joined view of events and their attributes. Events that do not have any +-- attributes are represented as a single row with empty key and value fields. +CREATE VIEW event_attributes AS + SELECT block_id, tx_id, type, key, composite_key, value + FROM events LEFT JOIN attributes ON (events.rowid = attributes.event_id); + +-- A joined view of all block events (those having tx_id NULL). +CREATE VIEW block_events AS + SELECT blocks.rowid as block_id, height, chain_id, type, key, composite_key, value + FROM blocks JOIN event_attributes ON (blocks.rowid = event_attributes.block_id) + WHERE event_attributes.tx_id IS NULL; + +-- A joined view of all transaction events. +CREATE VIEW tx_events AS + SELECT height, index, chain_id, type, key, composite_key, value, tx_results.created_at + FROM blocks JOIN tx_results ON (blocks.rowid = tx_results.block_id) + JOIN event_attributes ON (tx_results.rowid = event_attributes.tx_id) + WHERE event_attributes.tx_id IS NOT NULL; +``` + +The `PSQLEventSink` will implement the `EventSink` interface as follows +(some details omitted for brevity): + +```go +func NewEventSink(connStr, chainID string) (*EventSink, error) { + db, err := sql.Open(driverName, connStr) + // ... + + return &EventSink{ + store: db, + chainID: chainID, + }, nil +} + +func (es *EventSink) IndexBlockEvents(h types.EventDataNewBlockHeader) error { + ts := time.Now().UTC() + + return runInTransaction(es.store, func(tx *sql.Tx) error { + // Add the block to the blocks table and report back its row ID for use + // in indexing the events for the block. + blockID, err := queryWithID(tx, ` +INSERT INTO blocks (height, chain_id, created_at) + VALUES ($1, $2, $3) + ON CONFLICT DO NOTHING + RETURNING rowid; +`, h.Header.Height, es.chainID, ts) + // ... + + // Insert the special block meta-event for height. + if err := insertEvents(tx, blockID, 0, []abci.Event{ + makeIndexedEvent(types.BlockHeightKey, fmt.Sprint(h.Header.Height)), + }); err != nil { + return fmt.Errorf("block meta-events: %w", err) + } + // Insert all the block events. Order is important here, + if err := insertEvents(tx, blockID, 0, h.ResultBeginBlock.Events); err != nil { + return fmt.Errorf("begin-block events: %w", err) + } + if err := insertEvents(tx, blockID, 0, h.ResultEndBlock.Events); err != nil { + return fmt.Errorf("end-block events: %w", err) + } + return nil + }) +} + +func (es *EventSink) IndexTxEvents(txrs []*abci.TxResult) error { + ts := time.Now().UTC() + + for _, txr := range txrs { + // Encode the result message in protobuf wire format for indexing. + resultData, err := proto.Marshal(txr) + // ... + + // Index the hash of the underlying transaction as a hex string. + txHash := fmt.Sprintf("%X", types.Tx(txr.Tx).Hash()) + + if err := runInTransaction(es.store, func(tx *sql.Tx) error { + // Find the block associated with this transaction. + blockID, err := queryWithID(tx, ` +SELECT rowid FROM blocks WHERE height = $1 AND chain_id = $2; +`, txr.Height, es.chainID) + // ... + + // Insert a record for this tx_result and capture its ID for indexing events. + txID, err := queryWithID(tx, ` +INSERT INTO tx_results (block_id, index, created_at, tx_hash, tx_result) + VALUES ($1, $2, $3, $4, $5) + ON CONFLICT DO NOTHING + RETURNING rowid; +`, blockID, txr.Index, ts, txHash, resultData) + // ... + + // Insert the special transaction meta-events for hash and height. + if err := insertEvents(tx, blockID, txID, []abci.Event{ + makeIndexedEvent(types.TxHashKey, txHash), + makeIndexedEvent(types.TxHeightKey, fmt.Sprint(txr.Height)), + }); err != nil { + return fmt.Errorf("indexing transaction meta-events: %w", err) + } + // Index any events packaged with the transaction. + if err := insertEvents(tx, blockID, txID, txr.Result.Events); err != nil { + return fmt.Errorf("indexing transaction events: %w", err) + } + return nil + + }); err != nil { + return err + } + } + return nil +} + +// SearchBlockEvents is not implemented by this sink, and reports an error for all queries. +func (es *EventSink) SearchBlockEvents(ctx context.Context, q *query.Query) ([]int64, error) + +// SearchTxEvents is not implemented by this sink, and reports an error for all queries. +func (es *EventSink) SearchTxEvents(ctx context.Context, q *query.Query) ([]*abci.TxResult, error) + +// GetTxByHash is not implemented by this sink, and reports an error for all queries. +func (es *EventSink) GetTxByHash(hash []byte) (*abci.TxResult, error) + +// HasBlock is not implemented by this sink, and reports an error for all queries. +func (es *EventSink) HasBlock(h int64) (bool, error) +``` + +### Configuration + +The current `tx_index.indexer` configuration would be changed to accept a list +of supported `EventSink` types instead of a single value. + +Example: + +```toml +[tx_index] + +indexer = [ + "kv", + "psql" +] +``` + +If the `indexer` list contains the `null` indexer, then no indexers will be used +regardless of what other values may exist. + +Additional configuration parameters might be required depending on what event +sinks are supplied to `tx_index.indexer`. The `psql` will require an additional +connection configuration. + +```toml +[tx_index] + +indexer = [ + "kv", + "psql" +] + +pqsql_conn = "postgresql://:@:/?" +``` + +Any invalid or misconfigured `tx_index` configuration should yield an error as +early as possible. + +## Future Improvements + +Although not technically required to maintain feature parity with the current +existing Tendermint indexer, it would be beneficial for operators to have a method +of performing a "re-index". Specifically, Tendermint operators could invoke an +RPC method that allows the Tendermint node to perform a re-indexing of all block +and transaction events between two given heights, H1 and H2, +so long as the block store contains the blocks and transaction results for all +the heights specified in a given range. + +## Consequences + +### Positive + +- A more robust and flexible indexing and query engine for indexing and search + block and transaction events. +- The ability to not have to support a custom indexing and query engine beyond + the legacy `kv` type. +- The ability to offload/proxy indexing and querying to the underling sink. +- Scalability and reliability that essentially comes "for free" from the underlying + sink, if it supports it. + +### Negative + +- The need to support multiple and potentially a growing set of custom `EventSink` + types. + +### Neutral + +## References + +- [Cosmos SDK ADR-038](https://github.com/cosmos/cosmos-sdk/blob/master/docs/architecture/adr-038-state-listening.md) +- [PostgreSQL](https://www.postgresql.org/) +- [SQLite](https://www.sqlite.org/index.html) diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-066-e2e-testing.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-066-e2e-testing.md new file mode 100644 index 00000000..528e2523 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-066-e2e-testing.md @@ -0,0 +1,140 @@ +# ADR 66: End-to-End Testing + +## Changelog + +- 2020-09-07: Initial draft (@erikgrinaker) +- 2020-09-08: Minor improvements (@erikgrinaker) +- 2021-04-12: Renamed from RFC 001 (@tessr) + +## Authors + +- Erik Grinaker (@erikgrinaker) + +## Context + +The current set of end-to-end tests under `test/` are very limited, mostly focusing on P2P testing in a standard configuration. They do not test various configurations (e.g. fast sync reactor versions, state sync, block pruning, genesis vs InitChain setup), nor do they test various network topologies (e.g. sentry node architecture). This leads to poor test coverage, which has caused several serious bugs to go unnoticed. + +We need an end-to-end test suite that can run a large number of combinations of configuration options, genesis settings, network topologies, ABCI interactions, and failure scenarios and check that the network is still functional. This ADR outlines the basic requirements and design for such a system. + +This ADR will not cover comprehensive chaos testing, only a few simple scenarios (e.g. abrupt process termination and network partitioning). Chaos testing of the core consensus algorithm should be implemented e.g. via Jepsen tests or a similar framework, or alternatively be added to these end-to-end tests at a later time. Similarly, malicious or adversarial behavior is out of scope for the first implementation, but may be added later. + +## Proposal + +### Functional Coverage + +The following lists the functionality we would like to test: + +#### Environments + +- **Topology:** single node, 4 nodes (seeds and persistent), sentry architecture, NAT (UPnP) +- **Networking:** IPv4, IPv6 +- **ABCI connection:** UNIX socket, TCP, gRPC +- **PrivVal:** file, UNIX socket, TCP + +#### Node/App Configurations + +- **Database:** goleveldb, cleveldb, boltdb, rocksdb, badgerdb +- **Fast sync:** disabled, v0, v2 +- **State sync:** disabled, enabled +- **Block pruning:** none, keep 20, keep 1, keep random +- **Role:** validator, full node +- **App persistence:** enabled, disabled +- **Node modes:** validator, full, light, seed + +#### Geneses + +- **Validators:** none (InitChain), given +- **Initial height:** 1, 1000 +- **App state:** none, given + +#### Behaviors + +- **Recovery:** stop/start, power cycling, validator outage, network partition, total network loss +- **Validators:** add, remove, change power +- **Evidence:** injection of DuplicateVoteEvidence and LightClientAttackEvidence + +### Functional Combinations + +Running separate tests for all combinations of the above functionality is not feasible, as there are millions of them. However, the functionality can be grouped into three broad classes: + +- **Global:** affects the entire network, needing a separate testnet for each combination (e.g. topology, network protocol, genesis settings) + +- **Local:** affects a single node, and can be varied per node in a testnet (e.g. ABCI/privval connections, database backend, block pruning) + +- **Temporal:** can be run after each other in the same testnet (e.g. recovery and validator changes) + +Thus, we can run separate testnets for all combinations of global options (on the order of 100). In each testnet, we run nodes with randomly generated node configurations optimized for broad coverage (i.e. if one node is using GoLevelDB, then no other node should use it if possible). And in each testnet, we sequentially and randomly pick nodes to stop/start, power cycle, add/remove, disconnect, and so on. + +All of the settings should be specified in a testnet configuration (or alternatively the seed that generated it) such that it can be retrieved from CI and debugged locally. + +A custom ABCI application will have to be built that can exhibit the necessary behavior (e.g. make validator changes, prune blocks, enable/disable persistence, and so on). + +### Test Stages + +Given a test configuration, the test runner has the following stages: + +- **Setup:** configures the Docker containers and networks, but does not start them. + +- **Initialization:** starts the Docker containers, performs fast sync/state sync. Accomodates for different start heights. + +- **Perturbation:** adds/removes validators, restarts nodes, perturbs networking, etc - liveness and readiness checked between each operation. + +- **Testing:** runs RPC tests independently against all network nodes, making sure data matches expectations and invariants hold. + +### Tests + +The general approach will be to put the network through a sequence of operations (see stages above), check basic liveness and readiness after each operation, and then once the network stabilizes run an RPC test suite against each node in the network. + +The test suite will do black-box testing against a single node's RPC service. We will be testing the behavior of the network as a whole, e.g. that a fast synced node correctly catches up to the chain head and serves basic block data via RPC. Thus the tests will not send e.g. P2P messages or examine the node database, as these are considered internal implementation details - if the network behaves correctly, presumably the internal components function correctly. Comprehensive component testing (e.g. each and every RPC method parameter) should be done via unit/integration tests. + +The tests must take into account the node configuration (e.g. some nodes may be pruned, others may not be validators), and should somehow be provided access to expected data (i.e. complete block headers for the entire chain). + +The test suite should use the Tendermint RPC client and the Tendermint light client, to exercise the client code as well. + +### Implementation Considerations + +The testnets should run in Docker Compose, both locally and in CI. This makes it easier to reproduce test failures locally. Supporting multiple test-runners (e.g. on VMs or Kubernetes) is out of scope. The same image should be used for all tests, with configuration passed via a mounted volume. + +There does not appear to be any off-the-shelf solutions that would do this for us, so we will have to roll our own on top of Docker Compose. This gives us more flexibility, but is estimated to be a few weeks of work. + +Testnets should be configured via a YAML file. These are used as inputs for the test runner, which e.g. generates Docker Compose configurations from them. An additional layer on top should generate these testnet configurations from a YAML file that specifies all the option combinations to test. + +Comprehensive testnets should run against master nightly. However, a small subset of representative testnets should run for each pull request, e.g. a four-node IPv4 network with state sync and fast sync. + +Tests should be written using the standard Go test framework (and e.g. Testify), with a helper function to fetch info from the test configuration. The test runner will run the tests separately for each network node, and the test must vary its expectations based on the node's configuration. + +It should be possible to launch a specific testnet and run individual test cases from the IDE or local terminal against a it. + +If possible, the existing `testnet` command should be extended to set up the network topologies needed by the end-to-end tests. + +## Status + +Implemented + +## Consequences + +### Positive + +- Comprehensive end-to-end test coverage of basic Tendermint functionality, exercising common code paths in the same way that users would + +- Test environments can easily be reproduced locally and debugged via standard tooling + +### Negative + +- Limited coverage of consensus correctness testing (e.g. Jepsen) + +- No coverage of malicious or adversarial behavior + +- Have to roll our own test framework, which takes engineering resources + +- Possibly slower CI times, depending on which tests are run + +- Operational costs and overhead, e.g. infrastructure costs and system maintenance + +### Neutral + +- No support for alternative infrastructure platforms, e.g. Kubernetes or VMs + +## References + +- [#5291: new end-to-end test suite](https://github.com/tendermint/tendermint/issues/5291) diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-067-mempool-refactor.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-067-mempool-refactor.md new file mode 100644 index 00000000..d217b1df --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-067-mempool-refactor.md @@ -0,0 +1,303 @@ +# ADR 067: Mempool Refactor + +- [ADR 067: Mempool Refactor](#adr-067-mempool-refactor) + - [Changelog](#changelog) + - [Status](#status) + - [Context](#context) + - [Current Design](#current-design) + - [Alternative Approaches](#alternative-approaches) + - [Prior Art](#prior-art) + - [Ethereum](#ethereum) + - [Diem](#diem) + - [Decision](#decision) + - [Detailed Design](#detailed-design) + - [CheckTx](#checktx) + - [Mempool](#mempool) + - [Eviction](#eviction) + - [Gossiping](#gossiping) + - [Performance](#performance) + - [Future Improvements](#future-improvements) + - [Consequences](#consequences) + - [Positive](#positive) + - [Negative](#negative) + - [Neutral](#neutral) + - [References](#references) + +## Changelog + +- April 19, 2021: Initial Draft (@alexanderbez) + +## Status + +Accepted + +## Context + +Tendermint Core has a reactor and data structure, mempool, that facilitates the +ephemeral storage of uncommitted transactions. Honest nodes participating in a +Tendermint network gossip these uncommitted transactions to each other if they +pass the application's `CheckTx`. In addition, block proposers select from the +mempool a subset of uncommitted transactions to include in the next block. + +Currently, the mempool in Tendermint Core is designed as a FIFO queue. In other +words, transactions are included in blocks as they are received by a node. There +currently is no explicit and prioritized ordering of these uncommitted transactions. +This presents a few technical and UX challenges for operators and applications. + +Namely, validators are not able to prioritize transactions by their fees or any +incentive aligned mechanism. In addition, the lack of prioritization also leads +to cascading effects in terms of DoS and various attack vectors on networks, +e.g. [cosmos/cosmos-sdk#8224](https://github.com/cosmos/cosmos-sdk/discussions/8224). + +Thus, Tendermint Core needs the ability for an application and its users to +prioritize transactions in a flexible and performant manner. Specifically, we're +aiming to either improve, maintain or add the following properties in the +Tendermint mempool: + +- Allow application-determined transaction priority. +- Allow efficient concurrent reads and writes. +- Allow block proposers to reap transactions efficiently by priority. +- Maintain a fixed mempool capacity by transaction size and evict lower priority + transactions to make room for higher priority transactions. +- Allow transactions to be gossiped by priority efficiently. +- Allow operators to specify a maximum TTL for transactions in the mempool before + they're automatically evicted if not selected for a block proposal in time. +- Ensure the design allows for future extensions, such as replace-by-priority and + allowing multiple pending transactions per sender, to be incorporated easily. + +Note, not all of these properties will be addressed by the proposed changes in +this ADR. However, this proposal will ensure that any unaddressed properties +can be addressed in an easy and extensible manner in the future. + +### Current Design + +![mempool](./img/mempool-v0.jpeg) + +At the core of the `v0` mempool reactor is a concurrent linked-list. This is the +primary data structure that contains `Tx` objects that have passed `CheckTx`. +When a node receives a transaction from another peer, it executes `CheckTx`, which +obtains a read-lock on the `*CListMempool`. If the transaction passes `CheckTx` +locally on the node, it is added to the `*CList` by obtaining a write-lock. It +is also added to the `cache` and `txsMap`, both of which obtain their own respective +write-locks and map a reference from the transaction hash to the `Tx` itself. + +Transactions are continuously gossiped to peers whenever a new transaction is added +to a local node's `*CList`, where the node at the front of the `*CList` is selected. +Another transaction will not be gossiped until the `*CList` notifies the reader +that there are more transactions to gossip. + +When a proposer attempts to propose a block, they will execute `ReapMaxBytesMaxGas` +on the reactor's `*CListMempool`. This call obtains a read-lock on the `*CListMempool` +and selects as many transactions as possible starting from the front of the `*CList` +moving to the back of the list. + +When a block is finally committed, a caller invokes `Update` on the reactor's +`*CListMempool` with all the selected transactions. Note, the caller must also +explicitly obtain a write-lock on the reactor's `*CListMempool`. This call +will remove all the supplied transactions from the `txsMap` and the `*CList`, both +of which obtain their own respective write-locks. In addition, the transaction +may also be removed from the `cache` which obtains it's own write-lock. + +## Alternative Approaches + +When considering which approach to take for a priority-based flexible and +performant mempool, there are two core candidates. The first candidate is less +invasive in the required set of protocol and implementation changes, which +simply extends the existing `CheckTx` ABCI method. The second candidate essentially +involves the introduction of new ABCI method(s) and would require a higher degree +of complexity in protocol and implementation changes, some of which may either +overlap or conflict with the upcoming introduction of [ABCI++](https://github.com/tendermint/tendermint/blob/main/docs/rfc/rfc-013-abci%2B%2B.md). + +For more information on the various approaches and proposals, please see the +[mempool discussion](https://github.com/tendermint/tendermint/discussions/6295). + +## Prior Art + +### Ethereum + +The Ethereum mempool, specifically [Geth](https://github.com/ethereum/go-ethereum), +contains a mempool, `*TxPool`, that contains various mappings indexed by account, +such as a `pending` which contains all processable transactions for accounts +prioritized by nonce. It also contains a `queue` which is the exact same mapping +except it contains not currently processable transactions. The mempool also +contains a `priced` index of type `*txPricedList` that is a priority queue based +on transaction price. + +### Diem + +The [Diem mempool](https://github.com/diem/diem/blob/master/mempool/README.md#implementation-details) +contains a similar approach to the one we propose. Specifically, the Diem mempool +contains a mapping from `Account:[]Tx`. On top of this primary mapping from account +to a list of transactions, are various indexes used to perform certain actions. + +The main index, `PriorityIndex`. is an ordered queue of transactions that are +“consensus-ready” (i.e., they have a sequence number which is sequential to the +current sequence number for the account). This queue is ordered by gas price so +that if a client is willing to pay more (than other clients) per unit of +execution, then they can enter consensus earlier. + +## Decision + +To incorporate a priority-based flexible and performant mempool in Tendermint Core, +we will introduce new fields, `priority` and `sender`, into the `ResponseCheckTx` +type. + +We will introduce a new versioned mempool reactor, `v1` and assume an implicit +version of the current mempool reactor as `v0`. In the new `v1` mempool reactor, +we largely keep the functionality the same as `v0` except we augment the underlying +data structures. Specifically, we keep a mapping of senders to transaction objects. +On top of this mapping, we index transactions to provide the ability to efficiently +gossip and reap transactions by priority. + +## Detailed Design + +### CheckTx + +We introduce the following new fields into the `ResponseCheckTx` type: + +```diff +message ResponseCheckTx { + uint32 code = 1; + bytes data = 2; + string log = 3; // nondeterministic + string info = 4; // nondeterministic + int64 gas_wanted = 5 [json_name = "gas_wanted"]; + int64 gas_used = 6 [json_name = "gas_used"]; + repeated Event events = 7 [(gogoproto.nullable) = false, (gogoproto.jsontag) = "events,omitempty"]; + string codespace = 8; ++ int64 priority = 9; ++ string sender = 10; +} +``` + +It is entirely up the application in determining how these fields are populated +and with what values, e.g. the `sender` could be the signer and fee payer +of the transaction, the `priority` could be the cumulative sum of the fee(s). + +Only `sender` is required, while `priority` can be omitted which would result in +using the default value of zero. + +### Mempool + +The existing concurrent-safe linked-list will be replaced by a thread-safe map +of ``, i.e a mapping from `sender` to a single `*Tx` object, where +each `*Tx` is the next valid and processable transaction from the given `sender`. + +On top of this mapping, we index all transactions by priority using a thread-safe +priority queue, i.e. a [max heap](https://en.wikipedia.org/wiki/Min-max_heap). +When a proposer is ready to select transactions for the next block proposal, +transactions are selected from this priority index by highest priority order. +When a transaction is selected and reaped, it is removed from this index and +from the `` mapping. + +We define `Tx` as the following data structure: + +```go +type Tx struct { + // Tx represents the raw binary transaction data. + Tx []byte + + // Priority defines the transaction's priority as specified by the application + // in the ResponseCheckTx response. + Priority int64 + + // Sender defines the transaction's sender as specified by the application in + // the ResponseCheckTx response. + Sender string + + // Index defines the current index in the priority queue index. Note, if + // multiple Tx indexes are needed, this field will be removed and each Tx + // index will have its own wrapped Tx type. + Index int +} +``` + +### Eviction + +Upon successfully executing `CheckTx` for a new `Tx` and the mempool is currently +full, we must check if there exists a `Tx` of lower priority that can be evicted +to make room for the new `Tx` with higher priority and with sufficient size +capacity left. + +If such a `Tx` exists, we find it by obtaining a read lock and sorting the +priority queue index. Once sorted, we find the first `Tx` with lower priority and +size such that the new `Tx` would fit within the mempool's size limit. We then +remove this `Tx` from the priority queue index as well as the `` +mapping. + +This will require additional `O(n)` space and `O(n*log(n))` runtime complexity. Note that the space complexity does not depend on the size of the tx. + +### Gossiping + +We keep the existing thread-safe linked list as an additional index. Using this +index, we can efficiently gossip transactions in the same manner as they are +gossiped now (FIFO). + +Gossiping transactions will not require locking any other indexes. + +### Performance + +Performance should largely remain unaffected apart from the space overhead of +keeping an additional priority queue index and the case where we need to evict +transactions from the priority queue index. There should be no reads which +block writes on any index + +## Future Improvements + +There are a few considerable ways in which the proposed design can be improved or +expanded upon. Namely, transaction gossiping and for the ability to support +multiple transactions from the same `sender`. + +With regards to transaction gossiping, we need empirically validate whether we +need to gossip by priority. In addition, the current method of gossiping may not +be the most efficient. Specifically, broadcasting all the transactions a node +has in it's mempool to it's peers. Rather, we should explore for the ability to +gossip transactions on a request/response basis similar to Ethereum and other +protocols. Not only does this reduce bandwidth and complexity, but also allows +for us to explore gossiping by priority or other dimensions more efficiently. + +Allowing for multiple transactions from the same `sender` is important and will +most likely be a needed feature in the future development of the mempool, but for +now it suffices to have the preliminary design agreed upon. Having the ability +to support multiple transactions per `sender` will require careful thought with +regards to the interplay of the corresponding ABCI application. Regardless, the +proposed design should allow for adaptations to support this feature in a +non-contentious and backwards compatible manner. + +## Consequences + +### Positive + +- Transactions are allowed to be prioritized by the application. + +### Negative + +- Increased size of the `ResponseCheckTx` Protocol Buffer type. +- Causal ordering is NOT maintained. + - It is possible that certain transactions broadcasted in a particular order may + pass `CheckTx` but not end up being committed in a block because they fail + `CheckTx` later. e.g. Consider Tx1 that sends funds from existing + account Alice to a _new_ account Bob with priority P1 and then later + Bob's _new_ account sends funds back to Alice in Tx2 with P2, + such that P2 > P1. If executed in this order, both + transactions will pass `CheckTx`. However, when a proposer is ready to select + transactions for the next block proposal, they will select Tx2 before + Tx1 and thus Tx2 will _fail_ because Tx1 must + be executed first. This is because there is a _causal ordering_, + Tx1 ➝ Tx2. These types of situations should be rare as + most transactions are not causally ordered and can be circumvented by simply + trying again at a later point in time or by ensuring the "child" priority is + lower than the "parent" priority. In other words, if parents always have + priories that are higher than their children, then the new mempool design will + maintain causal ordering. + +### Neutral + +- A transaction that passed `CheckTx` and entered the mempool can later be evicted + at a future point in time if a higher priority transaction entered while the + mempool was full. + +## References + +- [ABCI++](https://github.com/tendermint/tendermint/blob/main/docs/rfc/rfc-013-abci%2B%2B.md) +- [Mempool Discussion](https://github.com/tendermint/tendermint/discussions/6295) diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-068-reverse-sync.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-068-reverse-sync.md new file mode 100644 index 00000000..d7ca5a91 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-068-reverse-sync.md @@ -0,0 +1,97 @@ +# ADR 068: Reverse Sync + +## Changelog + +- 20 April 2021: Initial Draft (@cmwaters) + +## Status + +Proposed + +## Context + +The advent of state sync and block pruning gave rise to the opportunity for full nodes to participate in consensus without needing complete block history. This also introduced a problem with respect to evidence handling. Nodes that didn't have all the blocks within the evidence age were incapable of validating evidence, thus halting if that evidence was committed on chain. + +[ADR 068](https://github.com/tendermint/tendermint/blob/main/docs/architecture/adr-068-reverse-sync.md) was published in response to this problem and modified the spec to add a minimum block history invariant. This predominantly sought to extend state sync so that it was capable of fetching and storing the `Header`, `Commit` and `ValidatorSet` (essentially a `LightBlock`) of the last `n` heights, where `n` was calculated based from the evidence age. + +This ADR sets out to describe the design of this state sync extension as well as modifications to the light client provider and the merging of tm store. + +## Decision + +The state sync reactor will be extended by introducing 2 new P2P messages (and a new channel). + +```protobuf +message LightBlockRequest { + uint64 height = 1; +} + +message LightBlockResponse { + tendermint.types.LightBlock light_block = 1; +} +``` + +This will be used by the "reverse sync" protocol that will fetch, verify and store prior light blocks such that the node can safely participate in consensus. + +Furthermore this allows for a new light client provider which offers the ability for the `StateProvider` to use the underlying P2P stack instead of RPC. + +## Detailed Design + +This section will focus first on the reverse sync (here we call it `backfill`) mechanism as a standalone protocol and then look to decribe how it integrates within the state sync reactor and how we define the new p2p light client provider. + +```go +// Backfill fetches, verifies, and stores necessary history +// to participate in consensus and validate evidence. +func (r *Reactor) backfill(state State) error {} +``` + +`State` is used to work out how far to go back, namely we need all light blocks that have: +- a height: `h >= state.LastBlockHeight - state.ConsensusParams.Evidence.MaxAgeNumBlocks` +- a time: `t >= state.LastBlockTime - state.ConsensusParams.Evidence.MaxAgeDuration` + +Reverse Sync relies on two components: A `Dispatcher` and a `BlockQueue`. The `Dispatcher` is a pattern taken from a similar [PR](https://github.com/tendermint/tendermint/pull/4508). It is wired to the `LightBlockChannel` and allows for concurrent light block requests by shifting through a linked list of peers. This abstraction has the nice quality that it can also be used as an array of light providers for a P2P based light client. + +The `BlockQueue` is a data structure that allows for multiple workers to fetch light blocks, serializing them for the main thread which picks them off the end of the queue, verifies the hashes and persists them. + +### Integration with State Sync + +Reverse sync is a blocking process that runs directly after syncing state and before transitioning into either fast sync or consensus. + +Prior, the state sync service was not connected to any db, instead it passed the state and commit back to the node. For reverse sync, state sync will be given access to both the `StateStore` and `BlockStore` to be able to write `Header`'s, `Commit`'s and `ValidatorSet`'s and read them so as to serve other state syncing peers. + +This also means adding new methods to these respective stores in order to persist them + +### P2P Light Client Provider + +As mentioned previously, the `Dispatcher` is capable of handling requests to multiple peers. We can therefore simply peel off a `blockProvider` instance which is assigned to each peer. By giving it the chain ID, the `blockProvider` is capable of doing a basic validation of the light block before returning it to the client. + +It's important to note that because state sync doesn't have access to the evidence channel it is incapable of allowing the light client to report evidence thus `ReportEvidence` is a no op. This is not too much of a concern for reverse sync but will need to be addressed for pure p2p light clients. + +### Pruning + +A final small note is with pruning. This ADR will introduce changes that will not allow an application to prune blocks that are within the evidence age. + +## Future Work + +This ADR tries to remain within the scope of extending state sync, however the changes made opens the door for several areas to be followed up: +- Properly integrate p2p messaging in the light client package. This will require adding the evidence channel so the light client is capable of reporting evidence. We may also need to rethink the providers model (i.e. currently providers are only added on start up) +- Merge and clean up the tendermint stores (state, block and evidence). This ADR adds new methods to both the state and block store for saving headers, commits and validator sets. This doesn't quite fit with the current struct (i.e. only `BlockMeta`s instead of `Header`s are saved). We should explore consolidating this for the sake of atomicity and the opportunity for batching. There are also other areas for changes such as the way we store block parts. See [here](https://github.com/tendermint/tendermint/issues/5383) and [here](https://github.com/tendermint/tendermint/issues/4630) for more context. +- Explore opportunistic reverse sync. Technically we don't need to reverse sync if no evidence is observed. I've tried to design the protocol such that it could be possible to move it across to the evidence package if we see fit. Thus only when evidence is seen where we don't have the necessary data, do we perform a reverse sync. The problem with this is that imagine we are in consensus and some evidence pops up requiring us to first fetch and verify the last 10,000 blocks. There's no way a node could do this (sequentially) and vote before the round finishes. Also as we don't punish invalid evidence, a malicious node could easily spam the chain just to get a bunch of "stateless" nodes to perform a bunch of useless work. +- Explore full reverse sync. Currently we only fetch light blocks. There might be benefits in the future to fetch and persist entire blocks especially if we give control to the application to do this. + +## Consequences + +### Positive + +- All nodes should have sufficient history to validate all types of evidence +- State syncing nodes can use the p2p layer for light client verification of state. This has better UX and could be faster but I haven't benchmarked. + +### Negative + +- Introduces more code = more maintenance + +### Neutral + +## References + +- [Reverse Sync RFC](https://github.com/tendermint/tendermint/blob/main/docs/architecture/adr-068-reverse-sync.md) +- [Original Issue](https://github.com/tendermint/tendermint/issues/5617) diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-069-flexible-node-initialization.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-069-flexible-node-initialization.md new file mode 100644 index 00000000..31082b10 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-069-flexible-node-initialization.md @@ -0,0 +1,268 @@ +# ADR 069: Flexible Node Initialization + +## Changlog + +- 2021-06-09: Initial Draft (@tychoish) + +- 2021-07-21: Major Revision (@tychoish) + +## Status + +Proposed. + +## Context + +In an effort to support [Go-API-Stability](./adr-060-go-api-stability.md), +during the 0.35 development cycle, we have attempted to reduce the the API +surface area by moving most of the interface of the `node` package into +unexported functions, as well as moving the reactors to an `internal` +package. Having this coincide with the 0.35 release made a lot of sense +because these interfaces were _already_ changing as a result of the `p2p` +[refactor](./adr-061-p2p-refactor-scope.md), so it made sense to think a bit +more about how tendermint exposes this API. + +While the interfaces of the P2P layer and most of the node package are already +internalized, this precludes some operational patterns that are important to +users who use tendermint as a library. Specifically, introspecting the +tendermint node service and replacing components is not supported in the latest +version of the code, and some of these use cases would require maintaining a +vendor copy of the code. Adding these features requires rather extensive +(internal/implementation) changes to the `node` and `rpc` packages, and this +ADR describes a model for changing the way that tendermint nodes initialize, in +service of providing this kind of functionality. + +We consider node initialization, because the current implemention +provides strong connections between all components, as well as between +the components of the node and the RPC layer, and being able to think +about the interactions of these components will help enable these +features and help define the requirements of the node package. + +## Alternative Approaches + +These alternatives are presented to frame the design space and to +contextualize the decision in terms of product requirements. These +ideas are not inherently bad, and may even be possible or desireable +in the (distant) future, and merely provide additional context for how +we, in the moment came to our decision(s). + +### Do Nothing + +The current implementation is functional and sufficient for the vast +majority of use cases (e.g., all users of the Cosmos-SDK as well as +anyone who runs tendermint and the ABCI application in separate +processes). In the current implementation, and even previous versions, +modifying node initialization or injecting custom components required +copying most of the `node` package, which required such users +to maintain a vendored copy of tendermint. + +While this is (likely) not tenable in the long term, as users do want +more modularity, and the current service implementation is brittle and +difficult to maintain, in the short term it may be possible to delay +implementation somewhat. Eventually, however, we will need to make the +`node` package easier to maintain and reason about. + +### Generic Service Pluggability + +One possible system design would export interfaces (in the Golang +sense) for all components of the system, to permit runtime dependency +injection of all components in the system, so that users can compose +tendermint nodes of arbitrary user-supplied components. + +Although this level of customization would provide benefits, it would be a huge +undertaking (particularly with regards to API design work) that we do not have +scope for at the moment. Eventually providing support for some kinds of +pluggability may be useful, so the current solution does not explicitly +foreclose the possibility of this alternative. + +### Abstract Dependency Based Startup and Shutdown + +The main proposal in this document makes tendermint node initialization simpler +and more abstract, but the system lacks a number of +features which daemon/service initialization could provide, such as a +system allowing the authors of services to control initialization and shutdown order +of components using dependency relationships. + +Such a system could work by allowing services to declare +initialization order dependencies to other reactors (by ID, perhaps) +so that the node could decide the initialization based on the +dependencies declared by services rather than requiring the node to +encode this logic directly. + +This level of configuration is probably more complicated than is needed. Given +that the authors of components in the current implementation of tendermint +already *do* need to know about other components, a dependency-based system +would probably be overly-abstract at this stage. + +## Decisions + +- To the greatest extent possible, factor the code base so that + packages are responsible for their own initialization, and minimize + the amount of code in the `node` package itself. + +- As a design goal, reduce direct coupling and dependencies between + components in the implementation of `node`. + +- Begin iterating on a more-flexible internal framework for + initializing tendermint nodes to make the initatilization process + less hard-coded by the implementation of the node objects. + + - Reactors should not need to expose their interfaces *within* the + implementation of the node type + + - This refactoring should be entirely opaque to users. + + - These node initialization changes should not require a + reevaluation of the `service.Service` or a generic initialization + orchestration framework. + +- Do not proactively provide a system for injecting + components/services within a tendtermint node, though make it + possible to retrofit this kind of plugability in the future if + needed. + +- Prioritize implementation of p2p-based statesync reactor to obviate + need for users to inject a custom state-sync provider. + +## Detailed Design + +The [current +nodeImpl](https://github.com/tendermint/tendermint/blob/main/node/node.go#L47) +includes direct references to the implementations of each of the +reactors, which should be replaced by references to `service.Service` +objects. This will require moving construction of the [rpc +service](https://github.com/tendermint/tendermint/blob/main/node/node.go#L771) +into the constructor of +[makeNode](https://github.com/tendermint/tendermint/blob/main/node/node.go#L126). One +possible implementation of this would be to eliminate the current +`ConfigureRPC` method on the node package and instead [configure it +here](https://github.com/tendermint/tendermint/pull/6798/files#diff-375d57e386f20eaa5f09f02bb9d28bfc48ac3dca18d0325f59492208219e5618R441). + +To avoid adding complexity to the `node` package, we will add a +composite service implementation to the `service` package +that implements `service.Service` and is composed of a sequence of +underlying `service.Service` objects and handles their +startup/shutdown in the specified sequential order. + +Consensus, blocksync (*née* fast sync), and statesync all depend on +each other, and have significant initialization dependencies that are +presently encoded in the `node` package. As part of this change, a +new package/component (likely named `blocks` located at +`internal/blocks`) will encapsulate the initialization of these block +management areas of the code. + +### Injectable Component Option + +This section briefly describes a possible implementation for +user-supplied services running within a node. This should not be +implemented unless user-supplied components are a hard requirement for +a user. + +In order to allow components to be replaced, a new public function +will be added to the public interface of `node` with a signature that +resembles the following: + +```go +func NewWithServices(conf *config.Config, + logger log.Logger, + cf proxy.ClientCreator, + gen *types.GenesisDoc, + srvs []service.Service, +) (service.Service, error) { +``` + +The `service.Service` objects will be initialized in the order supplied, after +all pre-configured/default services have started (and shut down in reverse +order). The given services may implement additional interfaces, allowing them +to replace specific default services. `NewWithServices` will validate input +service lists with the following rules: + +- None of the services may already be running. +- The caller may not supply more than one replacement reactor for a given + default service type. + +If callers violate any of these rules, `NewWithServices` will return +an error. To retract support for this kind of operation in the future, +the function can be modified to *always* return an error. + +## Consequences + +### Positive + +- The node package will become easier to maintain. + +- It will become easier to add additional services within tendermint + nodes. + +- It will become possible to replace default components in the node + package without vendoring the tendermint repo and modifying internal + code. + +- The current end-to-end (e2e) test suite will be able to prevent any + regressions, and the new functionality can be thoroughly unit tested. + +- The scope of this project is very narrow, which minimizes risk. + +### Negative + +- This increases our reliance on the `service.Service` interface which + is probably not an interface that we want to fully commit to. + +- This proposal implements a fairly minimal set of functionality and + leaves open the possibility for many additional features which are + not included in the scope of this proposal. + +### Neutral + +N/A + +## Open Questions + +- To what extent does this new initialization framework need to accommodate + the legacy p2p stack? Would it be possible to delay a great deal of this + work to the 0.36 cycle to avoid this complexity? + + - Answer: _depends on timing_, and the requirement to ship pluggable reactors in 0.35. + +- Where should additional public types be exported for the 0.35 + release? + + Related to the general project of API stabilization we want to deprecate + the `types` package, and move its contents into a new `pkg` hierarchy; + however, the design of the `pkg` interface is currently underspecified. + If `types` is going to remain for the 0.35 release, then we should consider + the impact of using multiple organizing modalities for this code within a + single release. + +## Future Work + +- Improve or simplify the `service.Service` interface. There are some + pretty clear limitations with this interface as written (there's no + way to timeout slow startup or shut down, the cycle between the + `service.BaseService` and `service.Service` implementations is + troubling, the default panic in `OnReset` seems troubling.) + +- As part of the refactor of `service.Service` have all services/nodes + respect the lifetime of a `context.Context` object, and avoid the + current practice of creating `context.Context` objects in p2p and + reactor code. This would be required for in-process multi-tenancy. + +- Support explicit dependencies between components and allow for + parallel startup, so that different reactors can startup at the same + time, where possible. + +## References + +- [the component + graph](https://peter.bourgon.org/go-for-industrial-programming/#the-component-graph) + as a framing for internal service construction. + +## Appendix + +### Dependencies + +There's a relationship between the blockchain and consensus reactor +described by the following dependency graph makes replacing some of +these components more difficult relative to other reactors or +components. + +![consensus blockchain dependency graph](./img/consensus_blockchain.png) diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-071-proposer-based-timestamps.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-071-proposer-based-timestamps.md new file mode 100644 index 00000000..e17226cc --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-071-proposer-based-timestamps.md @@ -0,0 +1,333 @@ +# ADR 71: Proposer-Based Timestamps + +## Changelog + + - July 15 2021: Created by @williambanfield + - Aug 4 2021: Draft completed by @williambanfield + - Aug 5 2021: Draft updated to include data structure changes by @williambanfield + - Aug 20 2021: Language edits completed by @williambanfield + - Oct 25 2021: Update the ADR to match updated spec from @cason by @williambanfield + - Nov 10 2021: Additional language updates by @williambanfield per feedback from @cason + - Feb 2 2022: Synchronize logic for timely with latest version of the spec by @williambanfield + +## Status + + **Accepted** + +## Context + +Tendermint currently provides a monotonically increasing source of time known as [BFTTime](https://github.com/tendermint/tendermint/blob/main/spec/consensus/bft-time.md). +This mechanism for producing a source of time is reasonably simple. +Each correct validator adds a timestamp to each `Precommit` message it sends. +The timestamp it sends is either the validator's current known Unix time or one millisecond greater than the previous block time, depending on which value is greater. +When a block is produced, the proposer chooses the block timestamp as the weighted median of the times in all of the `Precommit` messages the proposer received. +The weighting is proportional to the amount of voting power, or stake, a validator has on the network. +This mechanism for producing timestamps is both deterministic and byzantine fault tolerant. + +This current mechanism for producing timestamps has a few drawbacks. +Validators do not have to agree at all on how close the selected block timestamp is to their own currently known Unix time. +Additionally, any amount of voting power `>1/3` may directly control the block timestamp. +As a result, it is quite possible that the timestamp is not particularly meaningful. + +These drawbacks present issues in the Tendermint protocol. +Timestamps are used by light clients to verify blocks. +Light clients rely on correspondence between their own currently known Unix time and the block timestamp to verify blocks they see; +However, their currently known Unix time may be greatly divergent from the block timestamp as a result of the limitations of `BFTTime`. + +The proposer-based timestamps specification suggests an alternative approach for producing block timestamps that remedies these issues. +Proposer-based timestamps alter the current mechanism for producing block timestamps in two main ways: + +1. The block proposer is amended to offer up its currently known Unix time as the timestamp for the next block instead of the `BFTTime`. +1. Correct validators only approve the proposed block timestamp if it is close enough to their own currently known Unix time. + +The result of these changes is a more meaningful timestamp that cannot be controlled by `<= 2/3` of the validator voting power. +This document outlines the necessary code changes in Tendermint to implement the corresponding [proposer-based timestamps specification](https://github.com/tendermint/tendermint/tree/main/spec/consensus/proposer-based-timestamp). + +## Alternative Approaches + +### Remove timestamps altogether + +Computer clocks are bound to skew for a variety of reasons. +Using timestamps in our protocol means either accepting the timestamps as not reliable or impacting the protocol’s liveness guarantees. +This design requires impacting the protocol’s liveness in order to make the timestamps more reliable. +An alternate approach is to remove timestamps altogether from the block protocol. +`BFTTime` is deterministic but may be arbitrarily inaccurate. +However, having a reliable source of time is quite useful for applications and protocols built on top of a blockchain. + +We therefore decided not to remove the timestamp. +Applications often wish for some transactions to occur on a certain day, on a regular period, or after some time following a different event. +All of these require some meaningful representation of agreed upon time. +The following protocols and application features require a reliable source of time: +* Tendermint Light Clients [rely on correspondence between their known time](https://github.com/tendermint/tendermint/blob/main/spec/light-client/verification/README.md#definitions-1) and the block time for block verification. +* Tendermint Evidence validity is determined [either in terms of heights or in terms of time](https://github.com/tendermint/tendermint/blob/8029cf7a0fcc89a5004e173ec065aa48ad5ba3c8/spec/consensus/evidence.md#verification). +* Unbonding of staked assets in the Cosmos Hub [occurs after a period of 21 days](https://github.com/cosmos/governance/blob/ce75de4019b0129f6efcbb0e752cd2cc9e6136d3/params-change/Staking.md#unbondingtime). +* IBC packets can use either a [timestamp or a height to timeout packet delivery](https://docs.cosmos.network/v0.45/ibc/overview.html#acknowledgements) + +Finally, inflation distribution in the Cosmos Hub uses an approximation of time to calculate an annual percentage rate. +This approximation of time is calculated using [block heights with an estimated number of blocks produced in a year](https://github.com/cosmos/governance/blob/master/params-change/Mint.md#blocksperyear). +Proposer-based timestamps will allow this inflation calculation to use a more meaningful and accurate source of time. + + +## Decision + +Implement proposer-based timestamps and remove `BFTTime`. + +## Detailed Design + +### Overview + +Implementing proposer-based timestamps will require a few changes to Tendermint’s code. +These changes will be to the following components: +* The `internal/consensus/` package. +* The `state/` package. +* The `Vote`, `CommitSig` and `Header` types. +* The consensus parameters. + +### Changes to `CommitSig` + +The [CommitSig](https://github.com/tendermint/tendermint/blob/a419f4df76fe4aed668a6c74696deabb9fe73211/types/block.go#L604) struct currently contains a timestamp. +This timestamp is the current Unix time known to the validator when it issued a `Precommit` for the block. +This timestamp is no longer used and will be removed in this change. + +`CommitSig` will be updated as follows: + +```diff +type CommitSig struct { + BlockIDFlag BlockIDFlag `json:"block_id_flag"` + ValidatorAddress Address `json:"validator_address"` +-- Timestamp time.Time `json:"timestamp"` + Signature []byte `json:"signature"` +} +``` + +### Changes to `Vote` messages + +`Precommit` and `Prevote` messages use a common [Vote struct](https://github.com/tendermint/tendermint/blob/a419f4df76fe4aed668a6c74696deabb9fe73211/types/vote.go#L50). +This struct currently contains a timestamp. +This timestamp is set using the [voteTime](https://github.com/tendermint/tendermint/blob/e8013281281985e3ada7819f42502b09623d24a0/internal/consensus/state.go#L2241) function and therefore vote times correspond to the current Unix time known to the validator, provided this time is greater than the timestamp of the previous block. +For precommits, this timestamp is used to construct the [CommitSig that is included in the block in the LastCommit](https://github.com/tendermint/tendermint/blob/e8013281281985e3ada7819f42502b09623d24a0/types/block.go#L754) field. +For prevotes, this field is currently unused. +Proposer-based timestamps will use the timestamp that the proposer sets into the block and will therefore no longer require that a timestamp be included in the vote messages. +This timestamp is therefore no longer useful as part of consensus and may optionally be dropped from the message. + +`Vote` will be updated as follows: + +```diff +type Vote struct { + Type tmproto.SignedMsgType `json:"type"` + Height int64 `json:"height"` + Round int32 `json:"round"` + BlockID BlockID `json:"block_id"` // zero if vote is nil. +-- Timestamp time.Time `json:"timestamp"` + ValidatorAddress Address `json:"validator_address"` + ValidatorIndex int32 `json:"validator_index"` + Signature []byte `json:"signature"` +} +``` + +### New consensus parameters + +The proposer-based timestamp specification includes a pair of new parameters that must be the same among all validators. +These parameters are `PRECISION`, and `MSGDELAY`. + +The `PRECISION` and `MSGDELAY` parameters are used to determine if the proposed timestamp is acceptable. +A validator will only Prevote a proposal if the proposal timestamp is considered `timely`. +A proposal timestamp is considered `timely` if it is within `PRECISION` and `MSGDELAY` of the Unix time known to the validator. +More specifically, a proposal timestamp is `timely` if `proposalTimestamp - PRECISION ≤ validatorLocalTime ≤ proposalTimestamp + PRECISION + MSGDELAY`. + +Because the `PRECISION` and `MSGDELAY` parameters must be the same across all validators, they will be added to the [consensus parameters](https://github.com/tendermint/tendermint/blob/main/proto/tendermint/types/params.proto#L11) as [durations](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#google.protobuf.Duration). + +The consensus parameters will be updated to include this `Synchrony` field as follows: + +```diff +type ConsensusParams struct { + Block BlockParams `json:"block"` + Evidence EvidenceParams `json:"evidence"` + Validator ValidatorParams `json:"validator"` + Version VersionParams `json:"version"` +++ Synchrony SynchronyParams `json:"synchrony"` +} +``` + +```go +type SynchronyParams struct { + MessageDelay time.Duration `json:"message_delay"` + Precision time.Duration `json:"precision"` +} +``` + +### Changes to the block proposal step + +#### Proposer selects block timestamp + +Tendermint currently uses the `BFTTime` algorithm to produce the block's `Header.Timestamp`. +The [proposal logic](https://github.com/tendermint/tendermint/blob/68ca65f5d79905abd55ea999536b1a3685f9f19d/internal/state/state.go#L269) sets the weighted median of the times in the `LastCommit.CommitSigs` as the proposed block's `Header.Timestamp`. + +In proposer-based timestamps, the proposer will still set a timestamp into the `Header.Timestamp`. +The timestamp the proposer sets into the `Header` will change depending on if the block has previously received a [polka](https://github.com/tendermint/tendermint/blob/053651160f496bb44b107a434e3e6482530bb287/docs/introduction/what-is-tendermint.md#consensus-overview) or not. + +#### Proposal of a block that has not previously received a polka + +If a proposer is proposing a new block then it will set the Unix time currently known to the proposer into the `Header.Timestamp` field. +The proposer will also set this same timestamp into the `Timestamp` field of the `Proposal` message that it issues. + +#### Re-proposal of a block that has previously received a polka + +If a proposer is re-proposing a block that has previously received a polka on the network, then the proposer does not update the `Header.Timestamp` of that block. +Instead, the proposer simply re-proposes the exact same block. +This way, the proposed block has the exact same block ID as the previously proposed block and the validators that have already received that block do not need to attempt to receive it again. + +The proposer will set the re-proposed block's `Header.Timestamp` as the `Proposal` message's `Timestamp`. + +#### Proposer waits + +Block timestamps must be monotonically increasing. +In `BFTTime`, if a validator’s clock was behind, the [validator added 1 millisecond to the previous block’s time and used that in its vote messages](https://github.com/tendermint/tendermint/blob/e8013281281985e3ada7819f42502b09623d24a0/internal/consensus/state.go#L2246). +A goal of adding proposer-based timestamps is to enforce some degree of clock synchronization, so having a mechanism that completely ignores the Unix time of the validator time no longer works. +Validator clocks will not be perfectly in sync. +Therefore, the proposer’s current known Unix time may be less than the previous block's `Header.Time`. +If the proposer’s current known Unix time is less than the previous block's `Header.Time`, the proposer will sleep until its known Unix time exceeds it. + +This change will require amending the [defaultDecideProposal](https://github.com/tendermint/tendermint/blob/822893615564cb20b002dd5cf3b42b8d364cb7d9/internal/consensus/state.go#L1180) method. +This method should now schedule a timeout that fires when the proposer’s time is greater than the previous block's `Header.Time`. +When the timeout fires, the proposer will finally issue the `Proposal` message. + +### Changes to proposal validation rules + +The rules for validating a proposed block will be modified to implement proposer-based timestamps. +We will change the validation logic to ensure that a proposal is `timely`. + +Per the proposer-based timestamps spec, `timely` only needs to be checked if a block has not received a +2/3 majority of `Prevotes` in a round. +If a block previously received a +2/3 majority of prevotes in a previous round, then +2/3 of the voting power considered the block's timestamp near enough to their own currently known Unix time in that round. + +The validation logic will be updated to check `timely` for blocks that did not previously receive +2/3 prevotes in a round. +Receiving +2/3 prevotes in a round is frequently referred to as a 'polka' and we will use this term for simplicity. + +#### Current timestamp validation logic + +To provide a better understanding of the changes needed to timestamp validation, we will first detail how timestamp validation works currently in Tendermint. + +The [validBlock function](https://github.com/tendermint/tendermint/blob/c3ae6f5b58e07b29c62bfdc5715b6bf8ae5ee951/state/validation.go#L14) currently [validates the proposed block timestamp in three ways](https://github.com/tendermint/tendermint/blob/c3ae6f5b58e07b29c62bfdc5715b6bf8ae5ee951/state/validation.go#L118). +First, the validation logic checks that this timestamp is greater than the previous block’s timestamp. + +Second, it validates that the block timestamp is correctly calculated as the weighted median of the timestamps in the [block’s LastCommit](https://github.com/tendermint/tendermint/blob/e8013281281985e3ada7819f42502b09623d24a0/types/block.go#L48). + +Finally, the validation logic authenticates the timestamps in the `LastCommit.CommitSig`. +The cryptographic signature in each `CommitSig` is created by signing a hash of fields in the block with the voting validator’s private key. +One of the items in this `signedBytes` hash is the timestamp in the `CommitSig`. +To authenticate the `CommitSig` timestamp, the validator authenticating votes builds a hash of fields that includes the `CommitSig` timestamp and checks this hash against the signature. +This takes place in the [VerifyCommit function](https://github.com/tendermint/tendermint/blob/e8013281281985e3ada7819f42502b09623d24a0/types/validation.go#L25). + +#### Remove unused timestamp validation logic + +`BFTTime` validation is no longer applicable and will be removed. +This means that validators will no longer check that the block timestamp is a weighted median of `LastCommit` timestamps. +Specifically, we will remove the call to [MedianTime in the validateBlock function](https://github.com/tendermint/tendermint/blob/4db71da68e82d5cb732b235eeb2fd69d62114b45/state/validation.go#L117). +The `MedianTime` function can be completely removed. + +Since `CommitSig`s will no longer contain a timestamp, the validator authenticating a commit will no longer include the `CommitSig` timestamp in the hash of fields it builds to check against the cryptographic signature. + +#### Timestamp validation when a block has not received a polka + +The [POLRound](https://github.com/tendermint/tendermint/blob/68ca65f5d79905abd55ea999536b1a3685f9f19d/types/proposal.go#L29) in the `Proposal` message indicates which round the block received a polka. +A negative value in the `POLRound` field indicates that the block has not previously been proposed on the network. +Therefore the validation logic will check for timely when `POLRound < 0`. + +When a validator receives a `Proposal` message, the validator will check that the `Proposal.Timestamp` is at most `PRECISION` greater than the current Unix time known to the validator, and at maximum `PRECISION + MSGDELAY` less than the current Unix time known to the validator. +If the timestamp is not within these bounds, the proposed block will not be considered `timely`. + +Once a full block matching the `Proposal` message is received, the validator will also check that the timestamp in the `Header.Timestamp` of the block matches this `Proposal.Timestamp`. +Using the `Proposal.Timestamp` to check `timely` allows for the `MSGDELAY` parameter to be more finely tuned since `Proposal` messages do not change sizes and are therefore faster to gossip than full blocks across the network. + +A validator will also check that the proposed timestamp is greater than the timestamp of the block for the previous height. +If the timestamp is not greater than the previous block's timestamp, the block will not be considered valid, which is the same as the current logic. + +#### Timestamp validation when a block has received a polka + +When a block is re-proposed that has already received a +2/3 majority of `Prevote`s on the network, the `Proposal` message for the re-proposed block is created with a `POLRound` that is `>= 0`. +A validator will not check that the `Proposal` is `timely` if the propose message has a non-negative `POLRound`. +If the `POLRound` is non-negative, each validator will simply ensure that it received the `Prevote` messages for the proposed block in the round indicated by `POLRound`. + +If the validator does not receive `Prevote` messages for the proposed block before the proposal timeout, then it will prevote nil. +Validators already check that +2/3 prevotes were seen in `POLRound`, so this does not represent a change to the prevote logic. + +A validator will also check that the proposed timestamp is greater than the timestamp of the block for the previous height. +If the timestamp is not greater than the previous block's timestamp, the block will not be considered valid, which is the same as the current logic. + +Additionally, this validation logic can be updated to check that the `Proposal.Timestamp` matches the `Header.Timestamp` of the proposed block, but it is less relevant since checking that votes were received is sufficient to ensure the block timestamp is correct. + +#### Relaxation of the 'Timely' check + +The `Synchrony` parameters, `MessageDelay` and `Precision` provide a means to bound the timestamp of a proposed block. +Selecting values that are too small presents a possible liveness issue for the network. +If a Tendermint network selects a `MessageDelay` parameter that does not accurately reflect the time to broadcast a proposal message to all of the validators on the network, nodes will begin rejecting proposals from otherwise correct proposers because these proposals will appear to be too far in the past. + +`MessageDelay` and `Precision` are planned to be configured as `ConsensusParams`. +A very common way to update `ConsensusParams` is by executing a transaction included in a block that specifies new values for them. +However, if the network is unable to produce blocks because of this liveness issue, no such transaction may be executed. +To prevent this dangerous condition, we will add a relaxation mechanism to the `Timely` predicate. +If consensus takes more than 10 rounds to produce a block for any reason, the `MessageDelay` will be doubled. +This doubling will continue for each subsequent 10 rounds of consensus. +This will enable chains that selected too small of a value for the `MessageDelay` parameter to eventually issue a transaction and readjust the parameters to more accurately reflect the broadcast time. + +This liveness issue is not as problematic for chains with very small `Precision` values. +Operators can more easily readjust local validator clocks to be more aligned. +Additionally, chains that wish to increase a small `Precision` value can still take advantage of the `MessageDelay` relaxation, waiting for the `MessageDelay` value to grow significantly and issuing proposals with timestamps that are far in the past of their peers. + +For more discussion of this, see [issue 371](https://github.com/tendermint/spec/issues/371). + +### Changes to the prevote step + +Currently, a validator will prevote a proposal in one of three cases: + +* Case 1: Validator has no locked block and receives a valid proposal. +* Case 2: Validator has a locked block and receives a valid proposal matching its locked block. +* Case 3: Validator has a locked block, sees a valid proposal not matching its locked block but sees +2/3 prevotes for the proposal’s block, either in the current round or in a round greater than or equal to the round in which it locked its locked block. + +The only change we will make to the prevote step is to what a validator considers a valid proposal as detailed above. + +### Changes to the precommit step + +The precommit step will not require much modification. +Its proposal validation rules will change in the same ways that validation will change in the prevote step with the exception of the `timely` check: precommit validation will never check that the timestamp is `timely`. + +### Remove voteTime Completely + +[voteTime](https://github.com/tendermint/tendermint/blob/822893615564cb20b002dd5cf3b42b8d364cb7d9/internal/consensus/state.go#L2229) is a mechanism for calculating the next `BFTTime` given both the validator's current known Unix time and the previous block timestamp. +If the previous block timestamp is greater than the validator's current known Unix time, then voteTime returns a value one millisecond greater than the previous block timestamp. +This logic is used in multiple places and is no longer needed for proposer-based timestamps. +It should therefore be removed completely. + +## Future Improvements + +* Implement BLS signature aggregation. +By removing fields from the `Precommit` messages, we are able to aggregate signatures. + +## Consequences + +### Positive + +* `<2/3` of validators can no longer influence block timestamps. +* Block timestamp will have stronger correspondence to real time. +* Improves the reliability of light client block verification. +* Enables BLS signature aggregation. +* Enables evidence handling to use time instead of height for evidence validity. + +### Neutral + +* Alters Tendermint’s liveness properties. +Liveness now requires that all correct validators have synchronized clocks within a bound. +Liveness will now also require that validators’ clocks move forward, which was not required under `BFTTime`. + +### Negative + +* May increase the length of the propose step if there is a large skew between the previous proposer and the current proposer’s local Unix time. +This skew will be bound by the `PRECISION` value, so it is unlikely to be too large. + +* Current chains with block timestamps far in the future will either need to pause consensus until after the erroneous block timestamp or must maintain synchronized but very inaccurate clocks. + +## References + +* [PBTS Spec](https://github.com/tendermint/tendermint/tree/main/spec/consensus/proposer-based-timestamp) +* [BFTTime spec](https://github.com/tendermint/tendermint/blob/main/spec/consensus/bft-time.md) +* [Issue 371](https://github.com/tendermint/spec/issues/371) diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-072-request-for-comments.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-072-request-for-comments.md new file mode 100644 index 00000000..7b656d05 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-072-request-for-comments.md @@ -0,0 +1,105 @@ +# ADR 72: Restore Requests for Comments + +## Changelog + +- 20-Aug-2021: Initial draft (@creachadair) + +## Status + +Implemented + +## Context + +In the past, we kept a collection of Request for Comments (RFC) documents in `docs/rfc`. +Prior to the creation of the ADR process, these documents were used to document +design and implementation decisions about Tendermint Core. The RFC directory +was removed in favor of ADRs, in commit 3761aa69 (PR +[\#6345](https://github.com/tendermint/tendermint/pull/6345)). + +For issues where an explicit design decision or implementation change is +required, an ADR is generally preferable to an open-ended RFC: An ADR is +relatively narrowly-focused, identifies a specific design or implementation +question, and documents the consensus answer to that question. + +Some discussions are more open-ended, however, or don't require a specific +decision to be made (yet). Such conversations are still valuable to document, +and several members of the Tendermint team have been doing so by writing gists +or Google docs to share them around. That works well enough in the moment, but +gists do not support any kind of collaborative editing, and both gists and docs +are hard to discover after the fact. Google docs have much better collaborative +editing, but are worse for discoverability, especially when contributors span +different Google accounts. + +Discoverability is important, because these kinds of open-ended discussions are +useful to people who come later -- either as new team members or as outside +contributors seeking to use and understand the thoughts behind our designs and +the architectural decisions that arose from those discussion. + +With these in mind, I propose that: + +- We re-create a new, initially empty `docs/rfc` directory in the repository, + and use it to capture these kinds of open-ended discussions in supplement to + ADRs. + +- Unlike in the previous RFC scheme, documents in this new directory will + _not_ be used directly for decision-making. This is the key difference + between an RFC and an ADR. + + Instead, an RFC will exist to document background, articulate general + principles, and serve as a historical record of discussion and motivation. + + In this system, an RFC may _only_ result in a decision indirectly, via ADR + documents created in response to the RFC. + + **In short:** If a decision is required, write an ADR; otherwise if a + sufficiently broad discussion is needed, write an RFC. + +Just so that there is a consistent format, I also propose that: + +- RFC files are named `rfc-XXX-title.{md,rst,txt}` and are written in plain + text, Markdown, or ReStructured Text. + +- Like an ADR, an RFC should include a high-level change log at the top of the + document, and sections for: + + * Abstract: A brief, high-level synopsis of the topic. + * Background: Any background necessary to understand the topic. + * Discussion: Detailed discussion of the issue being considered. + +- Unlike an ADR, an RFC does _not_ include sections for Decisions, Detailed + Design, or evaluation of proposed solutions. If an RFC leads to a proposal + for an actual architectural change, that must be recorded in an ADR in the + usual way, and may refer back to the RFC in its References section. + +## Alternative Approaches + +Leaving aside implementation details, the main alternative to this proposal is +to leave things as they are now, with ADRs as the only log of record and other +discussions being held informally in whatever medium is convenient at the time. + +## Decision + +(pending) + +## Detailed Design + +- Create a new `docs/rfc` directory in the `tendermint` repository. Note that + this proposal intentionally does _not_ pull back the previous contents of + that path from Git history, as those documents were appropriately merged into + the ADR process. + +- Create a `README.md` for RFCs that explains the rules and their relationship + to ADRs. + +- Create an `rfc-template.md` file for RFC files. + +## Consequences + +### Positive + +- We will have a more discoverable place to record open-ended discussions that + do not immediately result in a design change. + +### Negative + +- Potentially some people could be confused about the RFC/ADR distinction. diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-073-libp2p.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-073-libp2p.md new file mode 100644 index 00000000..080fecbc --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-073-libp2p.md @@ -0,0 +1,235 @@ +# ADR 073: Adopt LibP2P + +## Changelog + +- 2021-11-02: Initial Draft (@tychoish) + +## Status + +Proposed. + +## Context + + +As part of the 0.35 development cycle, the Tendermint team completed +the first phase of the work described in ADRs 61 and 62, which included a +large scale refactoring of the reactors and the p2p message +routing. This replaced the switch and many of the other legacy +components without breaking protocol or network-level +interoperability and left the legacy connection/socket handling code. + +Following the release, the team has reexamined the state of the code +and the design, as well as Tendermint's requirements. The notes +from that process are available in the [P2P Roadmap +RFC][rfc]. + +This ADR supersedes the decisions made in ADRs 60 and 61, but +builds on the completed portions of this work. Previously, the +boundaries of peer management, message handling, and the higher level +business logic (e.g., "the reactors") were intermingled, and core +elements of the p2p system were responsible for the orchestration of +higher-level business logic. Refactoring the legacy components +made it more obvious that this entanglement of responsibilities +had outsized influence on the entire implementation, making +it difficult to iterate within the current abstractions. +It would not be viable to maintain interoperability with legacy +systems while also achieving many of our broader objectives. + +LibP2P is a thoroughly-specified implementation of a peer-to-peer +networking stack, designed specifically for systems such as +ours. Adopting LibP2P as the basis of Tendermint will allow the +Tendermint team to focus more of their time on other differentiating +aspects of the system, and make it possible for the ecosystem as a +whole to take advantage of tooling and efforts of the LibP2P +platform. + +## Alternative Approaches + +As discussed in the [P2P Roadmap RFC][rfc], the primary alternative would be to +continue development of Tendermint's home-grown peer-to-peer +layer. While that would give the Tendermint team maximal control +over the peer system, the current design is unexceptional on its +own merits, and the prospective maintenance burden for this system +exceeds our tolerances for the medium term. + +Tendermint can and should differentiate itself not on the basis of +its networking implementation or peer management tools, but providing +a consistent operator experience, a battle-tested consensus algorithm, +and an ergonomic user experience. + +## Decision + +Tendermint will adopt libp2p during the 0.37 development cycle, +replacing the bespoke Tendermint P2P stack. This will remove the +`Endpoint`, `Transport`, `Connection`, and `PeerManager` abstractions +and leave the reactors, `p2p.Router` and `p2p.Channel` +abstractions. + +LibP2P may obviate the need for a dedicated peer exchange (PEX) +reactor, which would also in turn obviate the need for a dedicated +seed mode. If this is the case, then all of this functionality would +be removed. + +If it turns out (based on the advice of Protocol Labs) that it makes +sense to maintain separate pubsub or gossipsub topics +per-message-type, then the `Router` abstraction could also +be entirely subsumed. + +## Detailed Design + +### Implementation Changes + +The seams in the P2P implementation between the higher level +constructs (reactors), the routing layer (`Router`) and the lower +level connection and peer management code make this operation +relatively straightforward to implement. A key +goal in this design is to minimize the impact on the reactors +(potentially entirely,) and completely remove the lower level +components (e.g., `Transport`, `Connection` and `PeerManager`) using the +separation afforded by the `Router` layer. The current state of the +code makes these changes relatively surgical, and limited to a small +number of methods: + +- `p2p.Router.OpenChannel` will still return a `Channel` structure + which will continue to serve as a pipe between the reactors and the + `Router`. The implementation will no longer need the queue + implementation, and will instead start goroutines that + are responsible for routing the messages from the channel to libp2p + fundamentals, replacing the current `p2p.Router.routeChannel`. + +- The current `p2p.Router.dialPeers` and `p2p.Router.acceptPeers`, + are responsible for establishing outbound and inbound connections, + respectively. These methods will be removed, along with + `p2p.Router.openConnection`, and the libp2p connection manager will + be responsible for maintaining network connectivity. + +- The `p2p.Channel` interface will change to replace Go + channels with a more functional interface for sending messages. + New methods on this object will take contexts to support safe + cancellation, and return errors, and will block rather than + running asynchronously. The `Out` channel through which + reactors send messages to Peers, will be replaced by a `Send` + method, and the Error channel will be replaced by an `Error` + method. + +- Reactors will be passed an interface that will allow them to + access Peer information from libp2p. This will supplant the + `p2p.PeerUpdates` subscription. + +- Add some kind of heartbeat message at the application level + (e.g. with a reactor,) potentially connected to libp2p's DHT to be + used by reactors for service discovery, message targeting, or other + features. + +- Replace the existing/legacy handshake protocol with [Noise](http://www.noiseprotocol.org/noise.html). + +This project will initially use the TCP-based transport protocols within +libp2p. QUIC is also available as an option that we may implement later. +We will not support mixed networks in the initial release, but will +revisit that possibility later if there is a demonstrated need. + +### Upgrade and Compatibility + +Because the routers and all current P2P libraries are `internal` +packages and not part of the public API, the only changes to the public +API surface area of Tendermint will be different configuration +file options, replacing the current P2P options with options relevant +to libp2p. + +However, it will not be possible to run a network with both networking +stacks active at once, so the upgrade to the version of Tendermint +will need to be coordinated between all nodes of the network. This is +consistent with the expectations around upgrades for Tendermint moving +forward, and will help manage both the complexity of the project and +the implementation timeline. + +## Open Questions + +- What is the role of Protocol Labs in the implementation of libp2p in + tendermint, both during the initial implementation and on an ongoing + basis thereafter? + +- Should all P2P traffic for a given node be pushed to a single topic, + so that a topic maps to a specific ChainID, or should + each reactor (or type of message) have its own topic? How many + topics can a libp2p network support? Is there testing that validates + the capabilities? + +- Tendermint presently provides a very coarse QoS-like functionality + using priorities based on message-type. + This intuitively/theoretically ensures that evidence and consensus + messages don't get starved by blocksync/statesync messages. It's + unclear if we can or should attempt to replicate this with libp2p. + +- What kind of QoS functionality does libp2p provide and what kind of + metrics does libp2p provide about it's QoS functionality? + +- Is it possible to store additional (and potentially arbitrary) + information into the DHT as part of the heartbeats between nodes, + such as the latest height, and then access that in the + reactors. How frequently can the DHT be updated? + +- Does it make sense to have reactors continue to consume inbound + messages from a Channel (`In`) or is there another interface or + pattern that we should consider? + + - We should avoid exposing Go channels when possible, and likely + some kind of alternate iterator likely makes sense for processing + messages within the reactors. + +- What are the security and protocol implications of tracking + information from peer heartbeats and exposing that to reactors? + +- How much (or how little) configuration can Tendermint provide for + libp2p, particularly on the first release? + + - In general, we should not support byo-functionality for libp2p + components within Tendermint, and reduce the configuration surface + area, as much as possible. + +- What are the best ways to provide request/response semantics for + reactors on top of libp2p? Will it be possible to add + request/response semantics in a future release or is there + anticipatory work that needs to be done as part of the initial + release? + +## Consequences + +### Positive + +- Reduce the maintenance burden for the Tendermint Core team by + removing a large swath of legacy code that has proven to be + difficult to modify safely. + +- Remove the responsibility for maintaining and developing the entire + peer management system (p2p) and stack. + +- Provide users with a more stable peer and networking system, + Tendermint can improve operator experience and network stability. + +### Negative + +- By deferring to library implementations for peer management and + networking, Tendermint loses some flexibility for innovating at the + peer and networking level. However, Tendermint should be innovating + primarily at the consensus layer, and libp2p does not preclude + optimization or development in the peer layer. + +- Libp2p is a large dependency and Tendermint would become dependent + upon Protocol Labs' release cycle and prioritization for bug + fixes. If this proves onerous, it's possible to maintain a vendor + fork of relevant components as needed. + +### Neutral + +- N/A + +## References + +- [ADR 61: P2P Refactor Scope][adr61] +- [ADR 62: P2P Architecture][adr62] +- [P2P Roadmap RFC][rfc] + +[adr61]: ./adr-061-p2p-refactor-scope.md +[adr62]: ./adr-062-p2p-architecture.md +[rfc]: ../rfc/rfc-000-p2p-roadmap.rst diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-074-timeout-params.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-074-timeout-params.md new file mode 100644 index 00000000..22fd784b --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-074-timeout-params.md @@ -0,0 +1,203 @@ +# ADR 74: Migrate Timeout Parameters to Consensus Parameters + +## Changelog + +- 03-Jan-2022: Initial draft (@williambanfield) +- 13-Jan-2022: Updated to indicate work on upgrade path needed (@williambanfield) + +## Status + +Proposed + +## Context + +### Background + +Tendermint's consensus timeout parameters are currently configured locally by each validator +in the validator's [config.toml][config-toml]. +This means that the validators on a Tendermint network may have different timeouts +from each other. There is no reason for validators on the same network to configure +different timeout values. Proper functioning of the Tendermint consensus algorithm +relies on these parameters being uniform across validators. + +The configurable values are as follows: + +* `TimeoutPropose` + * How long the consensus algorithm waits for a proposal block before issuing a prevote. + * If no prevote arrives by `TimeoutPropose`, then the consensus algorithm will issue a nil prevote. +* `TimeoutProposeDelta` + * How much the `TimeoutPropose` grows each round. +* `TimeoutPrevote` + * How long the consensus algorithm waits after receiving +2/3 prevotes with + no quorum for a value before issuing a precommit for nil. + (See the [arXiv paper][arxiv-paper], Algorithm 1, Line 34) +* `TimeoutPrevoteDelta` + * How much the `TimeoutPrevote` increases with each round. +* `TimeoutPrecommit` + * How long the consensus algorithm waits after receiving +2/3 precommits that + do not have a quorum for a value before entering the next round. + (See the [arXiv paper][arxiv-paper], Algorithm 1, Line 47) +* `TimeoutPrecommitDelta` + * How much the `TimeoutPrecommit` increases with each round. +* `TimeoutCommit` + * How long the consensus algorithm waits after committing a block but before starting the new height. + * This gives a validator a chance to receive slow precommits. +* `SkipTimeoutCommit` + * Make progress as soon as the node has 100% of the precommits. + + +### Overview of Change + +We will consolidate the timeout parameters and migrate them from the node-local +`config.toml` file into the network-global consensus parameters. + +The 8 timeout parameters will be consolidated down to 6. These will be as follows: + +* `TimeoutPropose` + * Same as current `TimeoutPropose`. +* `TimeoutProposeDelta` + * Same as current `TimeoutProposeDelta`. +* `TimeoutVote` + * How long validators wait for votes in both the prevote + and precommit phase of the consensus algorithm. This parameter subsumes + the current `TimeoutPrevote` and `TimeoutPrecommit` parameters. +* `TimeoutVoteDelta` + * How much the `TimeoutVote` will grow each successive round. + This parameter subsumes the current `TimeoutPrevoteDelta` and `TimeoutPrecommitDelta` + parameters. +* `TimeoutCommit` + * Same as current `TimeoutCommit`. +* `BypassCommitTimeout` + * Same as current `SkipTimeoutCommit`, renamed for clarity. + +A safe default will be provided by Tendermint for each of these parameters and +networks will be able to update the parameters as they see fit. Local updates +to these parameters will no longer be possible; instead, the application will control +updating the parameters. Applications using the Cosmos SDK will be automatically be +able to change the values of these consensus parameters [via a governance proposal][cosmos-sdk-consensus-params]. + +This change is low-risk. While parameters are locally configurable, many running chains +do not change them from their default values. For example, initializing +a node on Osmosis, Terra, and the Cosmos Hub using the their `init` command produces +a `config.toml` with Tendermint's default values for these parameters. + +### Why this parameter consolidation? + +Reducing the number of parameters is good for UX. Fewer superfluous parameters makes +running and operating a Tendermint network less confusing. + +The Prevote and Precommit messages are both similar sizes, require similar amounts +of processing so there is no strong need for them to be configured separately. + +The `TimeoutPropose` parameter governs how long Tendermint will wait for the proposed +block to be gossiped. Blocks are much larger than votes and therefore tend to be +gossiped much more slowly. It therefore makes sense to keep `TimeoutPropose` and +the `TimeoutProposeDelta` as parameters separate from the vote timeouts. + +`TimeoutCommit` is used by chains to ensure that the network waits for the votes from +slower validators before proceeding to the next height. Without this timeout, the votes +from slower validators would consistently not be included in blocks and those validators +would not be counted as 'up' from the chain's perspective. Being down damages a validator's +reputation and causes potential stakers to think twice before delegating to that validator. + +`TimeoutCommit` also prevents the network from producing the next height as soon as validators +on the fastest hardware with a summed voting power of +2/3 of the network's total have +completed execution of the block. Allowing the network to proceed as soon as the fastest ++2/3 completed execution would have a cumulative effect over heights, eventually +leaving slower validators unable to participate in consensus at all. `TimeoutCommit` +therefore allows networks to have greater variability in hardware. Additional +discussion of this can be found in [tendermint issue 5911][tendermint-issue-5911-comment] +and [spec issue 359][spec-issue-359]. + +## Alternative Approaches + +### Hardcode the parameters + +Many Tendermint networks run on similar cloud-hosted infrastructure. Therefore, +they have similar bandwidth and machine resources. The timings for propagating votes +and blocks are likely to be reasonably similar across networks. As a result, the +timeout parameters are good candidates for being hardcoded. Hardcoding the timeouts +in Tendermint would mean entirely removing these parameters from any configuration +that could be altered by either an application or a node operator. Instead, +Tendermint would ship with a set of timeouts and all applications using Tendermint +would use this exact same set of values. + +While Tendermint nodes often run with similar bandwidth and on similar cloud-hosted +machines, there are enough points of variability to make configuring +consensus timeouts meaningful. Namely, Tendermint network topologies are likely to be +very different from chain to chain. Additionally, applications may vary greatly in +how long the `Commit` phase may take. Applications that perform more work during `Commit` +require a longer `TimeoutCommit` to allow the application to complete its work +and be prepared for the next height. + +## Decision + +The decision has been made to implement this work, with the caveat that the +specific mechanism for introducing the new parameters to chains is still ongoing. + +## Detailed Design + +### New Consensus Parameters + +A new `TimeoutParams` `message` will be added to the [params.proto file][consensus-params-proto]. +This message will have the following form: + +```proto +message TimeoutParams { + google.protobuf.Duration propose = 1; + google.protobuf.Duration propose_delta = 2; + google.protobuf.Duration vote = 3; + google.protobuf.Duration vote_delta = 4; + google.protobuf.Duration commit = 5; + bool bypass_commit_timeout = 6; +} +``` + +This new message will be added as a field into the [`ConsensusParams` +message][consensus-params-proto]. The same default values that are [currently +set for these parameters][current-timeout-defaults] in the local configuration +file will be used as the defaults for these new consensus parameters in the +[consensus parameter defaults][default-consensus-params]. + +The new consensus parameters will be subject to the same +[validity rules][time-param-validation] as the current configuration values, +namely, each value must be non-negative. + +### Migration + +The new `ConsensusParameters` will be added during an upcoming release. In this +release, the old `config.toml` parameters will cease to control the timeouts and +an error will be logged on nodes that continue to specify these values. The specific +mechanism by which these parameters will added to a chain is being discussed in +[RFC-009][rfc-009] and will be decided ahead of the next release. + +The specific mechanism for adding these parameters depends on work related to +[soft upgrades][soft-upgrades], which is still ongoing. + +## Consequences + +### Positive + +* Timeout parameters will be equal across all of the validators in a Tendermint network. +* Remove superfluous timeout parameters. + +### Negative + +### Neutral + +* Timeout parameters require consensus to change. + +## References + +[conseusus-params-proto]: https://github.com/tendermint/spec/blob/a00de7199f5558cdd6245bbbcd1d8405ccfb8129/proto/tendermint/types/params.proto#L11 +[hashed-params]: https://github.com/tendermint/tendermint/blob/7cdf560173dee6773b80d1c574a06489d4c394fe/types/params.go#L49 +[default-consensus-params]: https://github.com/tendermint/tendermint/blob/7cdf560173dee6773b80d1c574a06489d4c394fe/types/params.go#L79 +[current-timeout-defaults]: https://github.com/tendermint/tendermint/blob/7cdf560173dee6773b80d1c574a06489d4c394fe/config/config.go#L955 +[config-toml]: https://github.com/tendermint/tendermint/blob/5cc980698a3402afce76b26693ab54b8f67f038b/config/toml.go#L425-L440 +[cosmos-sdk-consensus-params]: https://github.com/cosmos/cosmos-sdk/issues/6197 +[time-param-validation]: https://github.com/tendermint/tendermint/blob/7cdf560173dee6773b80d1c574a06489d4c394fe/config/config.go#L1038 +[tendermint-issue-5911-comment]: https://github.com/tendermint/tendermint/issues/5911#issuecomment-973560381 +[spec-issue-359]: https://github.com/tendermint/spec/issues/359 +[arxiv-paper]: https://arxiv.org/pdf/1807.04938.pdf +[soft-upgrades]: https://github.com/tendermint/spec/pull/222 +[rfc-009]: https://github.com/tendermint/tendermint/pull/7524 diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-075-rpc-subscription.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-075-rpc-subscription.md new file mode 100644 index 00000000..a838f276 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-075-rpc-subscription.md @@ -0,0 +1,684 @@ +# ADR 075: RPC Event Subscription Interface + +## Changelog + +- 01-Mar-2022: Update long-polling interface (@creachadair). +- 10-Feb-2022: Updates to reflect implementation. +- 26-Jan-2022: Marked accepted. +- 22-Jan-2022: Updated and expanded (@creachadair). +- 20-Nov-2021: Initial draft (@creachadair). + +--- +## Status + +Accepted + +--- +## Background & Context + +For context, see [RFC 006: Event Subscription][rfc006]. + +The [Tendermint RPC service][rpc-service] permits clients to subscribe to the +event stream generated by a consensus node. This allows clients to observe the +state of the consensus network, including details of the consensus algorithm +state machine, proposals, transaction delivery, and block completion. The +application may also attach custom key-value attributes to events to expose +application-specific details to clients. + +The event subscription API in the RPC service currently comprises three methods: + +1. `subscribe`: A request to subscribe to the events matching a specific + [query expression][query-grammar]. Events can be filtered by their key-value + attributes, including custom attributes provided by the application. + +2. `unsubscribe`: A request to cancel an existing subscription based on its + query expression. + +3. `unsubscribe_all`: A request to cancel all existing subscriptions belonging + to the client. + +There are some important technical and UX issues with the current RPC event +subscription API. The rest of this ADR outlines these problems in detail, and +proposes a new API scheme intended to address them. + +### Issue 1: Persistent connections + +To subscribe to a node's event stream, a client needs a persistent connection +to the node. Unlike the other methods of the service, for which each call is +serviced by a short-lived HTTP round trip, subscription delivers a continuous +stream of events to the client by hijacking the HTTP channel for a websocket. +The stream (and hence the HTTP request) persists until either the subscription +is explicitly canceled, or the connection is closed. + +There are several problems with this API: + +1. **Expensive per-connection state**: The server must maintain a substantial + amount of state per subscriber client: + + - The current implementation uses a [WebSocket][ws] for each active + subscriber. The connection must be maintained even if there are no + matching events for a given client. + + The server can drop idle connections to save resources, but doing so + terminates all subscriptions on those connections and forces those clients + to re-connect, adding additional resource churn for the server. + + - In addition, the server maintains a separate buffer of undelivered events + for each client. This is to reduce the dual risks that a client will miss + events, and that a slow client could "push back" on the publisher, + impeding the progress of consensus. + + Because event traffic is quite bursty, queues can potentially take up a + lot of memory. Moreover, each subscriber may have a different filter + query, so the server winds up having to duplicate the same events among + multiple subscriber queues. Not only does this add memory pressure, but it + does so most at the worst possible time, i.e., when the server is already + under load from high event traffic. + +2. **Operational access control is difficult**: The server's websocket + interface exposes _all_ the RPC service endpoints, not only the subscription + methods. This includes methods that allow callers to inject arbitrary + transactions (`broadcast_tx_*`) and evidence (`broadcast_evidence`) into the + network, remove transactions (`remove_tx`), and request arbitrary amounts of + chain state. + + Filtering requests to the GET endpoint is straightforward: A reverse proxy + like [nginx][nginx] can easily filter methods by URL path. Filtering POST + requests takes a bit more work, but can be managed with a filter program + that speaks [FastCGI][fcgi] and parses JSON-RPC request bodies. + + Filtering the websocket interface requires a dedicated proxy implementation. + Although nginx can [reverse-proxy websockets][rp-ws], it does not support + filtering websocket traffic via FastCGI. The operator would need to either + implement a custom [nginx extension module][ng-xm] or build and run a + standalone proxy that implements websocket and filters each session. Apart + from the work, this also makes the system even more resource intensive, as + well as introducing yet another connection that could potentially time out + or stall on full buffers. + + Even for the simple case of restricting access to only event subscription, + there is no easy solution currently: Once a caller has access to the + websocket endpoint, it has complete access to the RPC service. + +### Issue 2: Inconvenient client API + +The subscription interface has some inconvenient features for the client as +well as the server. These include: + +1. **Non-standard protocol:** The RPC service is mostly [JSON-RPC 2.0][jsonrpc2], + but the subscription interface diverges from the standard. + + In a standard JSON-RPC 2.0 call, the client initiates a request to the + server with a unique ID, and the server concludes the call by sending a + reply for that ID. The `subscribe` implementation, however, sends multiple + responses to the client's request: + + - The client sends `subscribe` with some ID `x` and the desired query + + - The server responds with ID `x` and an empty confirmation response. + + - The server then (repeatedly) sends event result responses with ID `x`, one + for each item with a matching event. + + Standard JSON-RPC clients will reject the subsequent replies, as they + announce a request ID (`x`) that is already complete. This means a caller + has to implement Tendermint-specific handling for these responses. + + Moreover, the result format is different between the initial confirmation + and the subsequent responses. This means a caller has to implement special + logic for decoding the first response versus the subsequent ones. + +2. **No way to detect data loss:** The subscriber connection can be terminated + for many reasons. Even ignoring ordinary network issues (e.g., packet loss): + + - The server will drop messages and/or close the websocket if its write + buffer fills, or if the queue of undelivered matching events is not + drained fast enough. The client has no way to discover that messages were + dropped even if the connection remains open. + + - Either the client or the server may close the websocket if the websocket + PING and PONG exchanges are not handled correctly, or frequently enough. + Even if correctly implemented, this may fail if the system is under high + load and cannot service those control messages in a timely manner. + + When the connection is terminated, the server drops all the subscriptions + for that client (as if it had called `unsubscribe_all`). Even if the client + reconnects, any events that were published during the period between the + disconnect and re-connect and re-subscription will be silently lost, and the + client has no way to discover that it missed some relevant messages. + +3. **No way to replay old events:** Even if a client knew it had missed some + events (due to a disconnection, for example), the API provides no way for + the client to "play back" events it may have missed. + +4. **Large response sizes:** Some event data can be quite large, and there can + be substantial duplication across items. The API allows the client to select + _which_ events are reported, but has no way to control which parts of a + matching event it wishes to receive. + + This can be costly on the server (which has to marshal those data into + JSON), the network, and the client (which has to unmarshal the result and + then pick through for the components that are relevant to it). + + Besides being inefficient, this also contributes to some of the persistent + connection issues mentioned above, e.g., filling up the websocket write + buffer and forcing the server to queue potentially several copies of a large + value in memory. + +5. **Client identity is tied to network address:** The Tendermint event API + identifies each subscriber by a (Client ID, Query) pair. In the RPC service, + the query is provided by the client, but the client ID is set to the TCP + address of the client (typically "host:port" or "ip:port"). + + This means that even if the server did _not_ drop subscriptions immediately + when the websocket connection is closed, a client may not be able to + reattach to its existing subscription. Dialing a new connection is likely + to result in a different port (and, depending on their own proxy setup, + possibly a different public IP). + + In isolation, this problem would be easy to work around with a new + subscription parameter, but it would require several other changes to the + handling of event subscriptions for that workaround to become useful. + +--- +## Decision + +To address the described problems, we will: + +1. Introduce a new API for event subscription to the Tendermint RPC service. + The proposed API is described in [Detailed Design](#detailed-design) below. + +2. This new API will target the Tendermint v0.36 release, during which the + current ("streaming") API will remain available as-is, but deprecated. + +3. The streaming API will be entirely removed in release v0.37, which will + require all users of event subscription to switch to the new API. + +> **Point for discussion:** Given that ABCI++ and PBTS are the main priorities +> for v0.36, it would be fine to slip the first phase of this work to v0.37. +> Unless there is a time problem, however, the proposed design does not disrupt +> the work on ABCI++ or PBTS, and will not increase the scope of breaking +> changes. Therefore the plan is to begin in v0.36 and slip only if necessary. + +--- +## Detailed Design + +### Design Goals + +Specific goals of this design include: + +1. Remove the need for a persistent connection to each subscription client. + Subscribers should use the same HTTP request flow for event subscription + requests as for other RPC calls. + +2. The server retains minimal state (possibly none) per-subscriber. In + particular: + + - The server does not buffer unconsumed writes nor queue undelivered events + on a per-client basis. + - A client that stalls or goes idle does not cost the server any resources. + - Any event data that is buffered or stored is shared among _all_ + subscribers, and is not duplicated per client. + +3. Slow clients have no impact (or minimal impact) on the rate of progress of + the consensus algorithm, beyond the ambient overhead of servicing individual + RPC requests. + +4. Clients can tell when they have missed events matching their subscription, + within some reasonable (configurable) window of time, and can "replay" + events within that window to catch up. + +5. Nice to have: It should be easy to use the event subscription API from + existing standard tools and libraries, including command-line use for + testing and experimentation. + +### Definitions + +- The **event stream** of a node is a single, time-ordered, heterogeneous + stream of event items. + +- Each **event item** comprises an **event datum** (for example, block header + metadata for a new-block event), and zero or more optional **events**. + +- An **event** means the [ABCI `Event` data type][abci-event], which comprises + a string type and zero or more string key-value **event attributes**. + + The use of the new terms "event item" and "event datum" is to avert confusion + between the values that are published to the event bus (what we call here + "event items") and the ABCI `Event` data type. + +- The node assigns each event item a unique identifier string called a + **cursor**. A cursor must be unique among all events published by a single + node, but it is not required to be unique globally across nodes. + + Cursors are time-ordered so that given event items A and B, if A was + published before B, then cursor(A) < cursor(B) in lexicographic order. + + A minimum viable cursor implementation is a tuple consisting of a timestamp + and a sequence number (e.g., `16CCC798FB5F4670-0123`). However, it may also + be useful to append basic type information to a cursor, to allow efficient + filtering (e.g., `16CCC87E91869050-0091:BeginBlock`). + + The initial implementation will use the minimum viable format. + +### Discussion + +The node maintains an **event log**, a shared ordered record of the events +published to its event bus within an operator-configurable time window. The +initial implementation will store the event log in-memory, and the operator +will be given two per-node configuration settings. Note, these names are +provisional: + +- `[rpc] event-log-window-size`: A duration before the latest published event, + during which the node will retain event items published. Setting this value + to zero disables event subscription. + +- `[rpc] event-log-max-items`: A maximum number of event items that the node + will retain within the time window. If the number of items exceeds this + value, the node discardes the oldest items in the window. Setting this value + to zero means that no limit is imposed on the number of items. + +The node will retain all events within the time window, provided they do not +exceed the maximum number. These config parameters allow the operator to +loosely regulate how much memory and storage the node allocates to the event +log. The client can use the server reply to tell whether the events it wants +are still available from the event log. + +The event log is shared among all subscribers to the node. + +> **Discussion point:** Should events persist across node restarts? +> +> The current event API does not persist events across restarts, so this new +> design does not either. Note, however, that we may "spill" older event data +> to disk as a way of controlling memory use. Such usage is ephemeral, however, +> and does not need to be tracked as node data (e.g., it could be temp files). + +### Query API + +To retrieve event data, the client will call the (new) RPC method `events`. +The parameters of this method will correspond to the following Go types: + +```go +type EventParams struct { + // Optional filter spec. If nil or empty, all items are eligible. + Filter *Filter `json:"filter"` + + // The maximum number of eligible results to return. + // If zero or negative, the server will report a default number. + MaxResults int `json:"max_results"` + + // Return only items after this cursor. If empty, the limit is just + // before the the beginning of the event log. + After string `json:"after"` + + // Return only items before this cursor. If empty, the limit is just + // after the head of the event log. + Before string `json:"before"` + + // Wait for up to this long for events to be available. + WaitTime time.Duration `json:"wait_time"` +} + +type Filter struct { + Query string `json:"query"` +} +``` + +> **Discussion point:** The initial implementation will not cache filter +> queries for the client. If this turns out to be a performance issue in +> production, the service can keep a small shared cache of compiled queries. +> Given the improvements from #7319 et seq., this should not be necessary. + +> **Discussion point:** For the initial implementation, the new API will use +> the existing query language as-is. Future work may extend the Filter message +> with a more structured and/or expressive query surface, but that is beyond +> the scope of this design. + +The semantics of the request are as follows: An item in the event log is +**eligible** for a query if: + +- It is newer than the `after` cursor (if set). +- It is older than the `before` cursor (if set). +- It matches the filter (if set). + +Among the eligible items in the log, the server returns up to `max_results` of +the newest items, in reverse order of cursor. If `max_results` is unset the +server chooses a number to return, and will cap `max_results` at a sensible +limit. + +The `wait_time` parameter is used to effect polling. If `before` is empty and +no items are available, the server will wait for up to `wait_time` for matching +items to arrive at the head of the log. If `wait_time` is zero or negative, the +server will wait for a default (positive) interval. + +If `before` non-empty, `wait_time` is ignored: new results are only added to +the head of the log, so there is no need to wait. This allows the client to +poll for new data, and "page" backward through matching event items. This is +discussed in more detail below. + +The server will set a sensible cap on the maximum `wait_time`, overriding +client-requested intervals longer than that. + +A successful reply from the `events` request corresponds to the following Go +types: + +```go +type EventReply struct { + // The items matching the request parameters, from newest + // to oldest, if any were available within the timeout. + Items []*EventItem `json:"items"` + + // This is true if there is at least one older matching item + // available in the log that was not returned. + More bool `json:"more"` + + // The cursor of the oldest item in the log at the time of this reply, + // or "" if the log is empty. + Oldest string `json:"oldest"` + + // The cursor of the newest item in the log at the time of this reply, + // or "" if the log is empty. + Newest string `json:"newest"` +} + +type EventItem struct { + // The cursor of this item. + Cursor string `json:"cursor"` + + // The encoded event data for this item. + // The type identifies the structure of the value. + Data struct { + Type string `json:"type"` + Value json.RawMessage `json:"value"` + } `json:"data"` +} +``` + +The `oldest` and `newest` fields of the reply report the cursors of the oldest +and newest items (of any kind) recorded in the event log at the time of the +reply, or are `""` if the log is empty. + +The `data` field contains the type-specific event datum. The datum carries any +ABCI events that may have been defined. + +> **Discussion point**: Based on [issue #7273][i7273], I did not include a +> separate field in the response for the ABCI events, since it duplicates data +> already stored elsewhere in the event data. + +The semantics of the reply are as follows: + +- If `items` is non-empty: + + - Items are ordered from newest to oldest. + + - If `more` is true, there is at least one additional, older item in the + event log that was not returned (in excess of `max_results`). + + In this case the client can fetch the next page by setting `before` in a + new request, to the cursor of the oldest item fetched (i.e., the last one + in `items`). + + - Otherwise (if `more` is false), all the matching results have been + reported (pagination is complete). + + - The first element of `items` identifies the newest item considered. + Subsequent poll requests can set `after` to this cursor to skip items + that were already retrieved. + +- If `items` is empty: + + - If the `before` was set in the request, there are no further eligible + items for this query in the log (pagination is complete). + + This is just a safety case; the client can detect this without issuing + another call by consulting the `more` field of the previous reply. + + - If the `before` was empty in the request, no eligible items were + available before the `wait_time` expired. The client may poll again to + wait for more event items. + +A client can store cursor values to detect data loss and to recover from +crashes and connectivity issues: + +- After a crash, the client requests events after the newest cursor it has + seen. If the reply indicates that cursor is no longer in range, the client + may (conservatively) conclude some event data may have been lost. + +- On the other hand, if it _is_ in range, the client can then page back through + the results that it missed, and then resume polling. As long as its recovery + cursor does not age out before it finishes, the client can be sure it has all + the relevant results. + +### Other Notes + +- The new API supports two general "modes" of operation: + + 1. In ordinary operation, clients will **long-poll** the head of the event + log for new events matching their criteria (by setting a `wait_time` and + no `before`). + + 2. If there are more events than the client requested, or if the client needs + to to read older events to recover from a stall or crash, clients will + **page** backward through the event log (by setting `before` and `after`). + +- While the new API requires explicit polling by the client, it makes better + use of the node's existing HTTP infrastructure (e.g., connection pools). + Moreover, the direct implementation is easier to use from standard tools and + client libraries for HTTP and JSON-RPC. + + Explicit polling does shift the burden of timeliness to the client. That is + arguably preferable, however, given that the RPC service is ancillary to the + node's primary goal, viz., consensus. The details of polling can be easily + hidden from client applications with simple libraries. + +- The format of a cursor is considered opaque to the client. Clients must not + parse cursor values, but they may rely on their ordering properties. + +- To maintain the event log, the server must prune items outside the time + window and in excess of the item limit. + + The initial implementation will do this by checking the tail of the event log + after each new item is published. If the number of items in the log exceeds + the item limit, it will delete oldest items until the log is under the limit; + then discard any older than the time window before the latest. + + To minimize coordination interference between the publisher (the event bus) + and the subcribers (the `events` service handlers), the event log will be + stored as a persistent linear queue with shared structure (a cons list). A + single reader-writer mutex will guard the "head" of the queue where new + items are published: + + - **To publish a new item**, the publisher acquires the write lock, conses a + new item to the front of the existing queue, and replaces the head pointer + with the new item. + + - **To scan the queue**, a reader acquires the read lock, captures the head + pointer, and then releases the lock. The rest of its request can be served + without holding a lock, since the queue structure will not change. + + When a reader wants to wait, it will yield the lock and wait on a condition + that is signaled when the publisher swings the pointer. + + - **To prune the queue**, the publisher (who is the sole writer) will track + the queue length and the age of the oldest item separately. When the + length and or age exceed the configured bounds, it will construct a new + queue spine on the same items, discarding out-of-band values. + + Pruning can be done while the publisher already holds the write lock, or + could be done outside the lock entirely: Once the new queue is constructed, + the lock can be re-acquired to swing the pointer. This costs some extra + allocations for the cons cells, but avoids duplicating any event items. + The pruning step is a simple linear scan down the first (up to) max-items + elements of the queue, to find the breakpoint of age and length. + + Moreover, the publisher can amortize the cost of pruning by item count, if + necessary, by pruning length "more aggressively" than the configuration + requires (e.g., reducing to 3/4 of the maximum rather than 1/1). + + The state of the event log before the publisher acquires the lock: + ![Before publish and pruning](./img/adr-075-log-before.png) + + After the publisher has added a new item and pruned old ones: + ![After publish and pruning](./img/adr-075-log-after.png) + +### Migration Plan + +This design requires that clients eventually migrate to the new event +subscription API, but provides a full release cycle with both APIs in place to +make this burden more tractable. The migration strategy is broadly: + +**Phase 1**: Release v0.36. + +- Implement the new `events` endpoint, keeping the existing methods as they are. +- Update the Go clients to support the new `events` endpoint, and handle polling. +- Update the old endpoints to log annoyingly about their own deprecation. +- Write tutorials about how to migrate client usage. + +At or shortly after release, we should proactively update the Cosmos SDK to use +the new API, to remove a disincentive to upgrading. + +**Phase 2**: Release v0.37 + +- During development, we should actively seek out any existing users of the + streaming event subscription API and help them migrate. +- Possibly also: Spend some time writing clients for JS, Rust, et al. +- Release: Delete the old implementation and all the websocket support code. + +> **Discussion point**: Even though the plan is to keep the existing service, +> we might take the opportunity to restrict the websocket endpoint to _only_ +> the event streaming service, removing the other endpoints. To minimize the +> disruption for users in the v0.36 cycle, I have decided not to do this for +> the first phase. +> +> If we wind up pushing this design into v0.37, however, we should re-evaulate +> this partial turn-down of the websocket. + +### Future Work + +- This design does not immediately address the problem of allowing the client + to control which data are reported back for event items. That concern is + deferred to future work. However, it would be straightforward to extend the + filter and/or the request parameters to allow more control. + +- The node currently stores a subset of event data (specifically the block and + transaction events) for use in reindexing. While these data are redundant + with the event log described in this document, they are not sufficient to + cover event subscription, as they omit other event types. + + In the future we should investigate consolidating or removing event data from + the state store entirely. For now this issue is out of scope for purposes of + updating the RPC API. We may be able to piggyback on the database unification + plans (see [RFC 001][rfc001]) to store the event log separately, so its + pruning policy does not need to be tied to the block and state stores. + +- This design reuses the existing filter query language from the old API. In + the future we may want to use a more structured and/or expressive query. The + Filter object can be extended with more fields as needed to support this. + +- Some users have trouble communicating with the RPC service because of + configuration problems like improperly-set CORS policies. While this design + does not address those issues directly, we might want to revisit how we set + policies in the RPC service to make it less susceptible to confusing errors + caused by misconfiguration. + +--- +## Consequences + +- ✅ Reduces the number of transport options for RPC. Supports [RFC 002][rfc002]. +- ️✅ Removes the primary non-standard use of JSON-RPC. +- ⛔️ Forces clients to migrate to a different API (eventually). +- ↕️ API requires clients to poll, but this reduces client state on the server. +- ↕️ We have to maintain both implementations for a whole release, but this + gives clients time to migrate. + +--- +## Alternative Approaches + +The following alternative approaches were considered: + +1. **Leave it alone.** Since existing tools mostly already work with the API as + it stands today, we could leave it alone and do our best to improve its + performance and reliability. + + Based on many issues reported by users and node operators (e.g., + [#3380][i3380], [#6439][i6439], [#6729][i6729], [#7247][i7247]), the + problems described here affect even the existing use that works. Investing + further incremental effort in the existing API is unlikely to address these + issues. + +2. **Design a better streaming API.** Instead of polling, we might try to + design a better "streaming" API for event subscription. + + A significant advantage of switching away from streaming is to remove the + need for persistent connections between the node and subscribers. A new + streaming protocol design would lose that advantage, and would still need a + way to let clients recover and replay. + + This approach might look better if we decided to use a different protocol + for event subscription, say gRPC instead of JSON-RPC. That choice, however, + would be just as breaking for existing clients, for marginal benefit. + Moreover, this option increases both the complexity and the resource cost on + the node implementation. + + Given that resource consumption and complexity are important considerations, + this option was not chosen. + +3. **Defer to an external event broker.** We might remove the entire event + subscription infrastructure from the node, and define an optional interface + to allow the node to publish all its events to an external event broker, + such as Apache Kafka. + + This has the advantage of greatly simplifying the node, but at a great cost + to the node operator: To enable event subscription in this design, the + operator has to stand up and maintain a separate process in communion with + the node, and configuration changes would have to be coordinated across + both. + + Moreover, this approach would be highly disruptive to existing client use, + and migration would probably require switching to third-party libraries. + Despite the potential benefits for the node itself, the costs to operators + and clients seems too large for this to be the best option. + + Publishing to an external event broker might be a worthwhile future project, + if there is any demand for it. That decision is out of scope for this design, + as it interacts with the design of the indexer as well. + +--- +## References + +- [RFC 006: Event Subscription][rfc006] +- [Tendermint RPC service][rpc-service] +- [Event query grammar][query-grammar] +- [RFC 6455: The WebSocket protocol][ws] +- [JSON-RPC 2.0 Specification][jsonrpc2] +- [Nginx proxy server][nginx] + - [Proxying websockets][rp-ws] + - [Extension modules][ng-xm] +- [FastCGI][fcgi] +- [RFC 001: Storage Engines & Database Layer][rfc001] +- [RFC 002: Interprocess Communication in Tendermint][rfc002] +- Issues: + - [rpc/client: test that client resubscribes upon disconnect][i3380] (#3380) + - [Too high memory usage when creating many events subscriptions][i6439] (#6439) + - [Tendermint emits events faster than clients can pull them][i6729] (#6729) + - [indexer: unbuffered event subscription slow down the consensus][i7247] (#7247) + - [rpc: remove duplication of events when querying][i7273] (#7273) + +[rfc006]: https://github.com/tendermint/tendermint/blob/main/docs/rfc/rfc-006-event-subscription.md +[rpc-service]: https://github.com/tendermint/tendermint/blob/main/rpc/openapi/openapi.yaml +[query-grammar]: https://pkg.go.dev/github.com/tendermint/tendermint@master/internal/pubsub/query/syntax +[ws]: https://datatracker.ietf.org/doc/html/rfc6455 +[jsonrpc2]: https://www.jsonrpc.org/specification +[nginx]: https://nginx.org/en/docs/ +[fcgi]: http://www.mit.edu/~yandros/doc/specs/fcgi-spec.html +[rp-ws]: https://nginx.org/en/docs/http/websocket.html + +[ng-xm]: https://www.nginx.com/resources/wiki/extending/ +[abci-event]: https://pkg.go.dev/github.com/tendermint/tendermint/abci/types#Event +[rfc001]: https://github.com/tendermint/tendermint/blob/main/docs/rfc/rfc-001-storage-engine.rst +[rfc002]: https://github.com/tendermint/tendermint/blob/main/docs/rfc/rfc-002-ipc-ecosystem.md +[i3380]: https://github.com/tendermint/tendermint/issues/3380 +[i6439]: https://github.com/tendermint/tendermint/issues/6439 +[i6729]: https://github.com/tendermint/tendermint/issues/6729 +[i7247]: https://github.com/tendermint/tendermint/issues/7247 +[i7273]: https://github.com/tendermint/tendermint/issues/7273 diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-076-combine-spec-repo.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-076-combine-spec-repo.md new file mode 100644 index 00000000..600bc631 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-076-combine-spec-repo.md @@ -0,0 +1,112 @@ +# ADR 076: Combine Spec and Tendermint Repositories + +## Changelog + +- 2022-02-04: Initial Draft. (@tychoish) + +## Status + +Implemented + +## Context + +While the specification for Tendermint was originally in the same +repository as the Go implementation, at some point the specification +was split from the core repository and maintained separately from the +implementation. While this makes sense in promoting a conceptual +separation of specification and implementation, in practice this +separation was a premature optimization, apparently aimed at supporting +alternate implementations of Tendermint. + +The operational and documentary burden of maintaining a separate +spec repo has not returned value to justify its cost. There are no active +projects to develop alternate implementations of Tendermint based on the +common specification, and having separate repositories creates an ongoing +burden to coordinate versions, documentation, and releases. + +## Decision + +The specification repository will be merged back into the Tendermint +core repository. + +Stakeholders including representatives from the maintainers of the +spec, the Go implementation, and the Tendermint Rust library, agreed +to merge the repositories in the Tendermint core dev meeting on 27 +January 2022, including @williambanfield @cmwaters @creachadair and +@thanethomson. + +## Alternative Approaches + +The main alternative we considered was to keep separate repositories, +and to introduce a coordinated versioning scheme between the two, so +that users could figure out which spec versions go with which versions +of the core implementation. + +We decided against this on the grounds that it would further complicate +the release process for _both_ repositories, without mitigating any of +the other existing issues. + +## Detailed Design + +Clone and merge the master branch of the `tendermint/spec` repository +as a branch of the `tendermint/tendermint`, to ensure the commit history +of both repositories remains intact. + +### Implementation Instructions + +1. Within the `tendermint` repository, execute the following commands + to add a new branch with the history of the master branch of `spec`: + + ```bash + git remote add spec git@github.com:tendermint/spec.git + git fetch spec + git checkout -b spec-master spec/master + mkdir spec + git ls-tree -z --name-only HEAD | xargs -0 -I {} git mv {} subdir/ + git commit -m "spec: organize specification prior to merge" + git checkout -b spec-merge-mainline origin/master + git merge --allow-unrelated-histories spec-master + ``` + + This merges the spec into the `tendermint/tendermint` repository as + a normal branch. This commit can also be backported to the 0.35 + branch, if needed. + +2. Migrate outstanding issues from `tendermint/spec` to the + `tendermint/tendermint` repository. + +3. In the specification repository, add redirect to the README and mark + the repository as archived. + + +## Consequences + +### Positive + +Easier maintenance for the specification will obviate a number of +complicated and annoying versioning problems, and will help prevent the +possibility of the specification and the implementation drifting apart. + +Additionally, co-locating the specification will help encourage +cross-pollination and collaboration, between engineers focusing on the +specification and the protocol and engineers focusing on the implementation. + +### Negative + +Co-locating the spec and Go implementation has the potential effect of +prioritizing the Go implementation with regards to the spec, and +making it difficult to think about alternate implementations of the +Tendermint algorithm. Although we may want to foster additional +Tendermint implementations in the future, this isn't an active goal +in our current roadmap, and *not* merging these repos doesn't +change the fact that the Go implementation of Tendermint is already the +primary implementation. + +### Neutral + +N/A + +## References + +- https://github.com/tendermint/tendermint/tree/main/spec +- https://github.com/tendermint/tendermint diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-077-block-retention.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-077-block-retention.md new file mode 100644 index 00000000..e44b24b3 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-077-block-retention.md @@ -0,0 +1,109 @@ +# ADR 077: Configurable Block Retention + +## Changelog + +- 2020-03-23: Initial draft (@erikgrinaker) +- 2020-03-25: Use local config for snapshot interval (@erikgrinaker) +- 2020-03-31: Use ABCI commit response for block retention hint +- 2020-04-02: Resolved open questions +- 2021-02-11: Migrate to tendermint repo (Originally [RFC 001](https://github.com/tendermint/spec/pull/84)) + +## Author(s) + +- Erik Grinaker (@erikgrinaker) + +## Context + +Currently, all Tendermint nodes contain the complete sequence of blocks from genesis up to some height (typically the latest chain height). This will no longer be true when the following features are released: + +- [Block pruning](https://github.com/tendermint/tendermint/issues/3652): removes historical blocks and associated data (e.g. validator sets) up to some height, keeping only the most recent blocks. + +- [State sync](https://github.com/tendermint/tendermint/issues/828): bootstraps a new node by syncing state machine snapshots at a given height, but not historical blocks and associated data. + +To maintain the integrity of the chain, the use of these features must be coordinated such that necessary historical blocks will not become unavailable or lost forever. In particular: + +- Some nodes should have complete block histories, for auditability, querying, and bootstrapping. + +- The majority of nodes should retain blocks longer than the Cosmos SDK unbonding period, for light client verification. + +- Some nodes must take and serve state sync snapshots with snapshot intervals less than the block retention periods, to allow new nodes to state sync and then replay blocks to catch up. + +- Applications may not persist their state on commit, and require block replay on restart. + +- Only a minority of nodes can be state synced within the unbonding period, for light client verification and to serve block histories for catch-up. + +However, it is unclear if and how we should enforce this. It may not be possible to technically enforce all of these without knowing the state of the entire network, but it may also be unrealistic to expect this to be enforced entirely through social coordination. This is especially unfortunate since the consequences of misconfiguration can be permanent chain-wide data loss. + +## Proposal + +Add a new field `retain_height` to the ABCI `ResponseCommit` message: + +```proto +service ABCIApplication { + rpc Commit(RequestCommit) returns (ResponseCommit); +} + +message RequestCommit {} + +message ResponseCommit { + // reserve 1 + bytes data = 2; // the Merkle root hash + uint64 retain_height = 3; // the oldest block height to retain +} +``` + +Upon ABCI `Commit`, which finalizes execution of a block in the state machine, Tendermint removes all data for heights lower than `retain_height`. This allows the state machine to control block retention, which is preferable since only it can determine the significance of historical blocks. By default (i.e. with `retain_height=0`) all historical blocks are retained. + +Removed data includes not only blocks, but also headers, commit info, consensus params, validator sets, and so on. In the first iteration this will be done synchronously, since the number of heights removed for each run is assumed to be small (often 1) in the typical case. It can be made asynchronous at a later time if this is shown to be necessary. + +Since `retain_height` is dynamic, it is possible for it to refer to a height which has already been removed. For example, commit at height 100 may return `retain_height=90` while commit at height 101 may return `retain_height=80`. This is allowed, and will be ignored - it is the application's responsibility to return appropriate values. + +State sync will eventually support backfilling heights, via e.g. a snapshot metadata field `backfill_height`, but in the initial version it will have a fully truncated block history. + +## Cosmos SDK Example + +As an example, we'll consider how the Cosmos SDK might make use of this. The specific details should be discussed in a separate SDK proposal. + +The returned `retain_height` would be the lowest height that satisfies: + +- Unbonding time: the time interval in which validators can be economically punished for misbehavior. Blocks in this interval must be auditable e.g. by the light client. + +- IAVL snapshot interval: the block interval at which the underlying IAVL database is persisted to disk, e.g. every 10000 heights. Blocks since the last IAVL snapshot must be available for replay on application restart. + +- State sync snapshots: blocks since the _oldest_ available snapshot must be available for state sync nodes to catch up (oldest because a node may be restoring an old snapshot while a new snapshot was taken). + +- Local config: archive nodes may want to retain more or all blocks, e.g. via a local config option `min-retain-blocks`. There may also be a need to vary rentention for other nodes, e.g. sentry nodes which do not need historical blocks. + +![Cosmos SDK block retention diagram](img/block-retention.png) + +## Status + +Implemented + +## Consequences + +### Positive + +- Application-specified block retention allows the application to take all relevant factors into account and prevent necessary blocks from being accidentally removed. + +- Node operators can independently decide whether they want to provide complete block histories (if local configuration for this is provided) and snapshots. + +### Negative + +- Social coordination is required to run archival nodes, failure to do so may lead to permanent loss of historical blocks. + +- Social coordination is required to run snapshot nodes, failure to do so may lead to inability to run state sync, and inability to bootstrap new nodes at all if no archival nodes are online. + +### Neutral + +- Reduced block retention requires application changes, and cannot be controlled directly in Tendermint. + +- Application-specified block retention may set a lower bound on disk space requirements for all nodes. + +## References + +- State sync ADR: + +- State sync issue: + +- Block pruning issue: diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-078-nonzero-genesis.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-078-nonzero-genesis.md new file mode 100644 index 00000000..39bb91e9 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-078-nonzero-genesis.md @@ -0,0 +1,82 @@ +# ADR 078: Non-Zero Genesis + +## Changelog + +- 2020-07-26: Initial draft (@erikgrinaker) +- 2020-07-28: Use weak chain linking, i.e. `predecessor` field (@erikgrinaker) +- 2020-07-31: Drop chain linking (@erikgrinaker) +- 2020-08-03: Add `State.InitialHeight` (@erikgrinaker) +- 2021-02-11: Migrate to tendermint repo (Originally [RFC 002](https://github.com/tendermint/spec/pull/119)) + +## Author(s) + +- Erik Grinaker (@erikgrinaker) + +## Context + +The recommended upgrade path for block protocol-breaking upgrades is currently to hard fork the +chain (see e.g. [`cosmoshub-3` upgrade](https://blog.cosmos.network/cosmos-hub-3-upgrade-announcement-39c9da941aee). +This is done by halting all validators at a predetermined height, exporting the application +state via application-specific tooling, and creating an entirely new chain using the exported +application state. + +As far as Tendermint is concerned, the upgraded chain is a completely separate chain, with e.g. +a new chain ID and genesis file. Notably, the new chain starts at height 1, and has none of the +old chain's block history. This causes problems for integrators, e.g. coin exchanges and +wallets, that assume a monotonically increasing height for a given blockchain. Users also find +it confusing that a given height can now refer to distinct states depending on the chain +version. + +An ideal solution would be to always retain block backwards compatibility in such a way that chain +history is never lost on upgrades. However, this may require a significant amount of engineering +work that is not viable for the planned Stargate release (Tendermint 0.34), and may prove too +restrictive for future development. + +As a first step, allowing the new chain to start from an initial height specified in the genesis +file would at least provide monotonically increasing heights. There was a proposal to include the +last block header of the previous chain as well, but since the genesis file is not verified and +hashed (only specific fields are) this would not be trustworthy. + +External tooling will be required to map historical heights onto e.g. archive nodes that contain +blocks from previous chain version. Tendermint will not include any such functionality. + +## Proposal + +Tendermint will allow chains to start from an arbitrary initial height: + +- A new field `initial_height` is added to the genesis file, defaulting to `1`. It can be set to any +non-negative integer, and `0` is considered equivalent to `1`. + +- A new field `InitialHeight` is added to the ABCI `RequestInitChain` message, with the same value +and semantics as the genesis field. + +- A new field `InitialHeight` is added to the `state.State` struct, where `0` is considered invalid. + Including the field here simplifies implementation, since the genesis value does not have to be + propagated throughout the code base separately, but it is not strictly necessary. + +ABCI applications may have to be updated to handle arbitrary initial heights, otherwise the initial +block may fail. + +## Status + +Implemented + +## Consequences + +### Positive + +- Heights can be unique throughout the history of a "logical" chain, across hard fork upgrades. + +### Negative + +- Upgrades still cause loss of block history. + +- Integrators will have to map height ranges to specific archive nodes/networks to query history. + +### Neutral + +- There is no explicit link to the last block of the previous chain. + +## References + +- [#2543: Allow genesis file to start from non-zero height w/ prev block header](https://github.com/tendermint/tendermint/issues/2543) diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-079-ed25519-verification.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-079-ed25519-verification.md new file mode 100644 index 00000000..81d69638 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-079-ed25519-verification.md @@ -0,0 +1,58 @@ +# ADR 079: Ed25519 Verification + +## Changelog + +- 2020-08-21: Initial RFC +- 2021-02-11: Migrate RFC to tendermint repo (Originally [RFC 003](https://github.com/tendermint/spec/pull/144)) + +## Author(s) + +- Marko (@marbar3778) + +## Context + +Ed25519 keys are the only supported key types for Tendermint validators currently. Tendermint-Go wraps the ed25519 key implementation from the go standard library. As more clients are implemented to communicate with the canonical Tendermint implementation (Tendermint-Go) different implementations of ed25519 will be used. Due to [RFC 8032](https://www.rfc-editor.org/rfc/rfc8032.html) not guaranteeing implementation compatibility, Tendermint clients must to come to an agreement of how to guarantee implementation compatibility. [Zcash](https://z.cash/) has multiple implementations of their client and have identified this as a problem as well. The team at Zcash has made a proposal to address this issue, [Zcash improvement proposal 215](https://zips.z.cash/zip-0215). + +## Proposal + +- Tendermint-Go would adopt [hdevalence/ed25519consensus](https://github.com/hdevalence/ed25519consensus). + - This library is implements `ed25519.Verify()` in accordance to zip-215. Tendermint-go will continue to use `crypto/ed25519` for signing and key generation. + +- Tendermint-rs would adopt [ed25519-zebra](https://github.com/ZcashFoundation/ed25519-zebra) + - related [issue](https://github.com/informalsystems/tendermint-rs/issues/355) + +Signature verification is one of the major bottlenecks of Tendermint-go, batch verification can not be used unless it has the same consensus rules, ZIP 215 makes verification safe in consensus critical areas. + +This change constitutes a breaking changes, therefore must be done in a major release. No changes to validator keys or operations will be needed for this change to be enabled. + +This change has no impact on signature aggregation. To enable this signature aggregation Tendermint will have to use different signature schema (Schnorr, BLS, ...). Secondly, this change will enable safe batch verification for the Tendermint-Go client. Batch verification for the rust client is already supported in the library being used. + +As part of the acceptance of this proposal it would be best to contract or discuss with a third party the process of conducting a security review of the go library. + +## Status + +Accepted (implicitly tracked in +[\#9186](https://github.com/tendermint/tendermint/issues/9186)) + +## Consequences + +### Positive + +- Consistent signature verification across implementations +- Enable safe batch verification + +### Negative + +#### Tendermint-Go + +- Third_party dependency + - library has not gone through a security review. + - unclear maintenance schedule +- Fragmentation of the ed25519 key for the go implementation, verification is done using a third party library while the rest + uses the go standard library + +### Neutral + +## References + +[It’s 255:19AM. Do you know what your validation criteria are?](https://hdevalence.ca/blog/2020-10-04-its-25519am) diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-080-reverse-sync.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-080-reverse-sync.md new file mode 100644 index 00000000..9e75dc88 --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-080-reverse-sync.md @@ -0,0 +1,203 @@ +# ADR 080: ReverseSync - fetching historical data + +## Changelog + +- 2021-02-11: Migrate to tendermint repo (Originally [RFC 005](https://github.com/tendermint/spec/pull/224)) +- 2021-04-19: Use P2P to gossip necessary data for reverse sync. +- 2021-03-03: Simplify proposal to the state sync case. +- 2021-02-17: Add notes on asynchronicity of processes. +- 2020-12-10: Rename backfill blocks to reverse sync. +- 2020-11-25: Initial draft. + +## Author(s) + +- Callum Waters (@cmwaters) + +## Context + +Two new features: [Block pruning](https://github.com/tendermint/tendermint/issues/3652) +and [State sync](https://github.com/tendermint/tendermint/blob/main/docs/architecture/adr-042-state-sync.md) +meant nodes no longer needed a complete history of the blockchain. This +introduced some challenges of its own which were covered and subsequently +tackled with [RFC-001](https://github.com/tendermint/tendermint/blob/main/docs/architecture/adr-077-block-retention.md). +The RFC allowed applications to set a block retention height; an upper bound on +what blocks would be pruned. However nodes who state sync past this upper bound +(which is necessary as snapshots must be saved within the trusting period for +the assisting light client to verify) have no means of backfilling the blocks +to meet the retention limit. This could be a problem as nodes who state sync and +then eventually switch to consensus (or fast sync) may not have the block and +validator history to verify evidence causing them to panic if they see 2/3 +commit on what the node believes to be an invalid block. + +Thus, this RFC sets out to instil a minimum block history invariant amongst +honest nodes. + +## Proposal + +A backfill mechanism can simply be defined as an algorithm for fetching, +verifying and storing, headers and validator sets of a height prior to the +current base of the node's blockchain. In matching the terminology used for +other data retrieving protocols (i.e. fast sync and state sync), we +call this method **ReverseSync**. + +We will define the mechanism in four sections: + +- Usage +- Design +- Verification +- Termination + +### Usage + +For now, we focus purely on the case of a state syncing node, whom after +syncing to a height will need to verify historical data in order to be capable +of processing new blocks. We can denote the earliest height that the node will +need to verify and store in order to be able to verify any evidence that might +arise as the `max_historical_height`/`time`. Both height and time are necessary +as this maps to the BFT time used for evidence expiration. After acquiring +`State`, we calculate these parameters as: + +```go +max_historical_height = max(state.InitialHeight, state.LastBlockHeight - state.ConsensusParams.EvidenceAgeHeight) +max_historical_time = max(GenesisTime, state.LastBlockTime.Sub(state.ConsensusParams.EvidenceAgeTime)) +``` + +Before starting either fast sync or consensus, we then run the following +synchronous process: + +```go +func ReverseSync(max_historical_height int64, max_historical_time time.Time) error +``` + +Where we fetch and verify blocks until a block `A` where +`A.Height <= max_historical_height` and `A.Time <= max_historical_time`. + +Upon successfully reverse syncing, a node can now safely continue. As this +feature is only used as part of state sync, one can think of this as merely an +extension to it. + +In the future we may want to extend this functionality to allow nodes to fetch +historical blocks for reasons of accountability or data accessibility. + +### Design + +This section will provide a high level overview of some of the more important +characteristics of the design, saving the more tedious details as an ADR. + +#### P2P + +Implementation of this RFC will require the addition of a new channel and two +new messages. + +```proto +message LightBlockRequest { + uint64 height = 1; +} +``` + +```proto +message LightBlockResponse { + Header header = 1; + Commit commit = 2; + ValidatorSet validator_set = 3; +} +``` + +The P2P path may also enable P2P networked light clients and a state sync that +also doesn't need to rely on RPC. + +### Verification + +ReverseSync is used to fetch the following data structures: + +- `Header` +- `Commit` +- `ValidatorSet` + +Nodes will also need to be able to verify these. This can be achieved by first +retrieving the header at the base height from the block store. From this trusted +header, the node hashes each of the three data structures and checks that they are correct. + +1. The trusted header's last block ID matches the hash of the new header + + ```go + header[height].LastBlockID == hash(header[height-1]) + ``` + +2. The trusted header's last commit hash matches the hash of the new commit + + ```go + header[height].LastCommitHash == hash(commit[height-1]) + ``` + +3. Given that the node now trusts the new header, check that the header's validator set + hash matches the hash of the validator set + + ```go + header[height-1].ValidatorsHash == hash(validatorSet[height-1]) + ``` + +### Termination + +ReverseSync draws a lot of parallels with fast sync. An important consideration +for fast sync that also extends to ReverseSync is termination. ReverseSync will +finish it's task when one of the following conditions have been met: + +1. It reaches a block `A` where `A.Height <= max_historical_height` and +`A.Time <= max_historical_time`. +2. None of it's peers reports to have the block at the height below the +processes current block. +3. A global timeout. + +This implies that we can't guarantee adequate history and thus the term +"invariant" can't be used in the strictest sense. In the case that the first +condition isn't met, the node will log an error and optimistically attempt +to continue with either fast sync or consensus. + +## Alternative Solutions + +The need for a minimum block history invariant stems purely from the need to +validate evidence (although there may be some application relevant needs as +well). Because of this, an alternative, could be to simply trust whatever the +2/3+ majority has agreed upon and in the case where a node is at the head of the +blockchain, you simply abstain from voting. + +As it stands, if 2/3+ vote on evidence you can't verify, in the same manner if +2/3+ vote on a header that a node sees as invalid (perhaps due to a different +app hash), the node will halt. + +Another alternative is the method with which the relevant data is retrieved. +Instead of introducing new messages to the P2P layer, RPC could have been used +instead. + +The aforementioned data is already available via the following RPC endpoints: +`/commit` for `Header`'s' and `/validators` for `ValidatorSet`'s'. It was +decided predominantly due to the instability of the current RPC infrastructure +that P2P be used instead. + +## Status + +Proposed + +## Consequences + +### Positive + +- Ensures a minimum block history invariant for honest nodes. This will allow + nodes to verify evidence. + +### Negative + +- Statesync will be slower as more processing is required. + +### Neutral + +- By having validator sets served through p2p, this would make it easier to +extend p2p support to light clients and state sync. +- In the future, it may also be possible to extend this feature to allow for +nodes to freely fetch and verify prior blocks + +## References + +- [RFC-001: Block retention](https://github.com/tendermint/tendermint/blob/main/docs/architecture/adr-077-block-retention.md) +- [Original issue](https://github.com/tendermint/tendermint/issues/4629) diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/adr-081-protobuf-mgmt.md b/cometbft/v0.38/docs/architecture/tendermint-core/adr-081-protobuf-mgmt.md new file mode 100644 index 00000000..23d734ef --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/adr-081-protobuf-mgmt.md @@ -0,0 +1,201 @@ +# ADR 081: Protocol Buffers Management + +## Changelog + +- 2022-02-28: First draft + +## Status + +Accepted + +[Tracking issue](https://github.com/tendermint/tendermint/issues/8121) + +## Context + +At present, we manage the [Protocol Buffers] schema files ("protos") that define +our wire-level data formats within the Tendermint repository itself (see the +[`proto`](../../proto/) directory). Recently, we have been making use of [Buf], +both locally and in CI, in order to generate Go stubs, and lint and check +`.proto` files for breaking changes. + +The version of Buf used at the time of this decision was `v1beta1`, and it was +discussed in [\#7975] and in weekly calls as to whether we should upgrade to +`v1` and harmonize our approach with that used by the Cosmos SDK. The team +managing the Cosmos SDK was primarily interested in having our protos versioned +and easily accessible from the [Buf] registry. + +The three main sets of stakeholders for the `.proto` files and their needs, as +currently understood, are as follows. + +1. Tendermint needs Go code generated from `.proto` files. +2. Consumers of Tendermint's `.proto` files, specifically projects that want to + interoperate with Tendermint and need to generate code for their own + programming language, want to be able to access these files in a reliable and + efficient way. +3. The Tendermint Core team wants to provide stable interfaces that are as easy + as possible to maintain, on which consumers can depend, and to be able to + notify those consumers promptly when those interfaces change. To this end, we + want to: + 1. Prevent any breaking changes from being introduced in minor/patch releases + of Tendermint. Only major version updates should be able to contain + breaking interface changes. + 2. Prevent generated code from diverging from the Protobuf schema files. + +There was also discussion surrounding the notion of automated documentation +generation and hosting, but it is not clear at this time whether this would be +that valuable to any of our stakeholders. What will, of course, be valuable at +minimum would be better documentation (in comments) of the `.proto` files +themselves. + +## Alternative Approaches + +### Meeting stakeholders' needs + +1. Go stub generation from protos. We could use: + 1. [Buf]. This approach has been rather cumbersome up to this point, and it + is not clear what Buf really provides beyond that which `protoc` provides + to justify the additional complexity in configuring Buf for stub + generation. + 2. [protoc] - the Protocol Buffers compiler. +2. Notification of breaking changes: + 1. Buf in CI for all pull requests to *release* branches only (and not on + `master`). + 2. Buf in CI on every pull request to every branch (this was the case at the + time of this decision, and the team decided that the signal-to-noise ratio + for this approach was too low to be of value). +3. `.proto` linting: + 1. Buf in CI on every pull request +4. `.proto` formatting: + 1. [clang-format] locally and a [clang-format GitHub Action] in CI to check + that files are formatted properly on every pull request. +5. Sharing of `.proto` files in a versioned, reliable manner: + 1. Consumers could simply clone the Tendermint repository, check out a + specific commit, tag or branch and manually copy out all of the `.proto` + files they need. This requires no effort from the Tendermint Core team and + will continue to be an option for consumers. The drawback of this approach + is that it requires manual coding/scripting to implement and is brittle in + the face of bigger changes. + 2. Uploading our `.proto` files to Buf's registry on every release. This is + by far the most seamless for consumers of our `.proto` files, but requires + the dependency on Buf. This has the additional benefit that the Buf + registry will automatically [generate and host + documentation][buf-docs-gen] for these protos. + 3. We could create a process that, upon release, creates a `.zip` file + containing our `.proto` files. + +### Popular alternatives to Buf + +[Prototool] was not considered as it appears deprecated, and the ecosystem seems +to be converging on Buf at this time. + +### Tooling complexity + +The more tools we have in our build/CI processes, the more complex and fragile +repository/CI management becomes, and the longer it takes to onboard new team +members. Maintainability is a core concern here. + +### Buf sustainability and costs + +One of the primary considerations regarding the usage of Buf is whether, for +example, access to its registry will eventually become a +paid-for/subscription-based service and whether this is valuable enough for us +and the ecosystem to pay for such a service. At this time, it appears as though +Buf will never charge for hosting open source projects' protos. + +Another consideration was Buf's sustainability as a project - what happens when +their resources run out? Will there be a strong and broad enough open source +community to continue maintaining it? + +### Local Buf usage options + +Local usage of Buf (i.e. not in CI) can be accomplished in two ways: + +1. Installing the relevant tools individually. +2. By way of its [Docker image][buf-docker]. + +Local installation of Buf requires developers to manually keep their toolchains +up-to-date. The Docker option comes with a number of complexities, including +how the file system permissions of code generated by a Docker container differ +between platforms (e.g. on Linux, Buf-generated code ends up being owned by +`root`). + +The trouble with the Docker-based approach is that we make use of the +[gogoprotobuf] plugin for `protoc`. Continuing to use the Docker-based approach +to using Buf will mean that we will have to continue building our own custom +Docker image with embedded gogoprotobuf. + +Along these lines, we could eventually consider coming up with a [Nix]- or +[redo]-based approach to developer tooling to ensure tooling consistency across +the team and for anyone who wants to be able to contribute to Tendermint. + +## Decision + +1. We will adopt Buf for now for proto generation, linting, breakage checking + and its registry (mainly in CI, with optional usage locally). +2. Failing CI when checking for breaking changes in `.proto` files will only + happen when performing minor/patch releases. +3. Local tooling will be favored over Docker-based tooling. + +## Detailed Design + +We currently aim to: + +1. Update to Buf `v1` to facilitate linting, breakage checking and uploading to + the Buf registry. +2. Configure CI appropriately for proto management: + 1. Uploading protos to the Buf registry on every release (e.g. the + [approach][cosmos-sdk-buf-registry-ci] used by the Cosmos SDK). + 2. Linting on every pull request (e.g. the + [approach][cosmos-sdk-buf-linting-ci] used by the Cosmos SDK). The linter + passing should be considered a requirement for accepting PRs. + 3. Checking for breaking changes in minor/patch version releases and failing + CI accordingly - see [\#8003]. + 4. Add [clang-format GitHub Action] to check `.proto` file formatting. Format + checking should be considered a requirement for accepting PRs. +3. Update the Tendermint [`Makefile`](../../Makefile) to primarily facilitate + local Protobuf stub generation, linting, formatting and breaking change + checking. More specifically: + 1. This includes removing the dependency on Docker and introducing the + dependency on local toolchain installation. CI-based equivalents, where + relevant, will rely on specific GitHub Actions instead of the Makefile. + 2. Go code generation will rely on `protoc` directly. + +## Consequences + +### Positive + +- We will still offer Go stub generation, proto linting and breakage checking. +- Breakage checking will only happen on minor/patch releases to increase the + signal-to-noise ratio in CI. +- Versioned protos will be made available via Buf's registry upon every release. + +### Negative + +- Developers/contributors will need to install the relevant Protocol + Buffers-related tooling (Buf, gogoprotobuf, clang-format) locally in order to + build, lint, format and check `.proto` files for breaking changes. + +### Neutral + +## References + +- [Protocol Buffers] +- [Buf] +- [\#7975] +- [protoc] - The Protocol Buffers compiler + +[Protocol Buffers]: https://developers.google.com/protocol-buffers +[Buf]: https://buf.build/ +[\#7975]: https://github.com/tendermint/tendermint/pull/7975 +[protoc]: https://github.com/protocolbuffers/protobuf +[clang-format]: https://clang.llvm.org/docs/ClangFormat.html +[clang-format GitHub Action]: https://github.com/marketplace/actions/clang-format-github-action +[buf-docker]: https://hub.docker.com/r/bufbuild/buf +[cosmos-sdk-buf-registry-ci]: https://github.com/cosmos/cosmos-sdk/blob/e6571906043b6751951a42b6546431b1c38b05bd/.github/workflows/proto-registry.yml +[cosmos-sdk-buf-linting-ci]: https://github.com/cosmos/cosmos-sdk/blob/e6571906043b6751951a42b6546431b1c38b05bd/.github/workflows/proto.yml#L15 +[\#8003]: https://github.com/tendermint/tendermint/issues/8003 +[Nix]: https://nixos.org/ +[gogoprotobuf]: https://github.com/cosmos/gogoproto +[Prototool]: https://github.com/uber/prototool +[buf-docs-gen]: https://docs.buf.build/bsr/documentation +[redo]: https://redo.readthedocs.io/en/latest/ diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/img/adr-046-fig1.png b/cometbft/v0.38/docs/architecture/tendermint-core/img/adr-046-fig1.png new file mode 100644 index 00000000..d68712e8 Binary files /dev/null and b/cometbft/v0.38/docs/architecture/tendermint-core/img/adr-046-fig1.png differ diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/img/adr-062-architecture.svg b/cometbft/v0.38/docs/architecture/tendermint-core/img/adr-062-architecture.svg new file mode 100644 index 00000000..4a824eee --- /dev/null +++ b/cometbft/v0.38/docs/architecture/tendermint-core/img/adr-062-architecture.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/img/adr-075-log-after.png b/cometbft/v0.38/docs/architecture/tendermint-core/img/adr-075-log-after.png new file mode 100644 index 00000000..359f205e Binary files /dev/null and b/cometbft/v0.38/docs/architecture/tendermint-core/img/adr-075-log-after.png differ diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/img/adr-075-log-before.png b/cometbft/v0.38/docs/architecture/tendermint-core/img/adr-075-log-before.png new file mode 100644 index 00000000..813b9d25 Binary files /dev/null and b/cometbft/v0.38/docs/architecture/tendermint-core/img/adr-075-log-before.png differ diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/img/bc-reactor-refactor.png b/cometbft/v0.38/docs/architecture/tendermint-core/img/bc-reactor-refactor.png new file mode 100644 index 00000000..4cd84a02 Binary files /dev/null and b/cometbft/v0.38/docs/architecture/tendermint-core/img/bc-reactor-refactor.png differ diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/img/bc-reactor.png b/cometbft/v0.38/docs/architecture/tendermint-core/img/bc-reactor.png new file mode 100644 index 00000000..f7fe0f81 Binary files /dev/null and b/cometbft/v0.38/docs/architecture/tendermint-core/img/bc-reactor.png differ diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/img/bifurcation-point.png b/cometbft/v0.38/docs/architecture/tendermint-core/img/bifurcation-point.png new file mode 100644 index 00000000..dce8938d Binary files /dev/null and b/cometbft/v0.38/docs/architecture/tendermint-core/img/bifurcation-point.png differ diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/img/block-retention.png b/cometbft/v0.38/docs/architecture/tendermint-core/img/block-retention.png new file mode 100644 index 00000000..e013e1ab Binary files /dev/null and b/cometbft/v0.38/docs/architecture/tendermint-core/img/block-retention.png differ diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/img/blockchain-reactor-v1.png b/cometbft/v0.38/docs/architecture/tendermint-core/img/blockchain-reactor-v1.png new file mode 100644 index 00000000..70debcd6 Binary files /dev/null and b/cometbft/v0.38/docs/architecture/tendermint-core/img/blockchain-reactor-v1.png differ diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/img/blockchain-reactor-v2.png b/cometbft/v0.38/docs/architecture/tendermint-core/img/blockchain-reactor-v2.png new file mode 100644 index 00000000..5f15333a Binary files /dev/null and b/cometbft/v0.38/docs/architecture/tendermint-core/img/blockchain-reactor-v2.png differ diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/img/blockchain-v2-channels.png b/cometbft/v0.38/docs/architecture/tendermint-core/img/blockchain-v2-channels.png new file mode 100644 index 00000000..69886da9 Binary files /dev/null and b/cometbft/v0.38/docs/architecture/tendermint-core/img/blockchain-v2-channels.png differ diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/img/consensus_blockchain.png b/cometbft/v0.38/docs/architecture/tendermint-core/img/consensus_blockchain.png new file mode 100644 index 00000000..dd0f4daa Binary files /dev/null and b/cometbft/v0.38/docs/architecture/tendermint-core/img/consensus_blockchain.png differ diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/img/evidence_lifecycle.png b/cometbft/v0.38/docs/architecture/tendermint-core/img/evidence_lifecycle.png new file mode 100644 index 00000000..dc4ed54f Binary files /dev/null and b/cometbft/v0.38/docs/architecture/tendermint-core/img/evidence_lifecycle.png differ diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/img/formula1.png b/cometbft/v0.38/docs/architecture/tendermint-core/img/formula1.png new file mode 100644 index 00000000..447ee30f Binary files /dev/null and b/cometbft/v0.38/docs/architecture/tendermint-core/img/formula1.png differ diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/img/formula2.png b/cometbft/v0.38/docs/architecture/tendermint-core/img/formula2.png new file mode 100644 index 00000000..081a1576 Binary files /dev/null and b/cometbft/v0.38/docs/architecture/tendermint-core/img/formula2.png differ diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/img/light-client-detector.png b/cometbft/v0.38/docs/architecture/tendermint-core/img/light-client-detector.png new file mode 100644 index 00000000..b098aa6e Binary files /dev/null and b/cometbft/v0.38/docs/architecture/tendermint-core/img/light-client-detector.png differ diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/img/mempool-v0.jpeg b/cometbft/v0.38/docs/architecture/tendermint-core/img/mempool-v0.jpeg new file mode 100644 index 00000000..dc3da6e7 Binary files /dev/null and b/cometbft/v0.38/docs/architecture/tendermint-core/img/mempool-v0.jpeg differ diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/img/pbts-message.png b/cometbft/v0.38/docs/architecture/tendermint-core/img/pbts-message.png new file mode 100644 index 00000000..400f3569 Binary files /dev/null and b/cometbft/v0.38/docs/architecture/tendermint-core/img/pbts-message.png differ diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/img/state-sync.png b/cometbft/v0.38/docs/architecture/tendermint-core/img/state-sync.png new file mode 100644 index 00000000..08b6eac4 Binary files /dev/null and b/cometbft/v0.38/docs/architecture/tendermint-core/img/state-sync.png differ diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/img/tags1.png b/cometbft/v0.38/docs/architecture/tendermint-core/img/tags1.png new file mode 100644 index 00000000..a6bc64e8 Binary files /dev/null and b/cometbft/v0.38/docs/architecture/tendermint-core/img/tags1.png differ diff --git a/cometbft/v0.38/docs/architecture/tendermint-core/img/tm-amnesia-attack.png b/cometbft/v0.38/docs/architecture/tendermint-core/img/tm-amnesia-attack.png new file mode 100644 index 00000000..7e084b27 Binary files /dev/null and b/cometbft/v0.38/docs/architecture/tendermint-core/img/tm-amnesia-attack.png differ diff --git a/cometbft/v0.38/docs/core/RPC.mdx b/cometbft/v0.38/docs/core/RPC.mdx new file mode 100644 index 00000000..36de5ab1 --- /dev/null +++ b/cometbft/v0.38/docs/core/RPC.mdx @@ -0,0 +1,15 @@ +--- +order: 9 +--- + +# RPC + +The RPC documentation is hosted here: + +- [OpenAPI reference](../rpc) + +{/* +NOTE: The OpenAPI reference (../rpc) is injected into the documentation during +the CometBFT docs build process. See https://github.com/cometbft/cometbft-docs/ +for details. +*/} diff --git a/cometbft/v0.38/docs/core/Running-in-production.mdx b/cometbft/v0.38/docs/core/Running-in-production.mdx new file mode 100644 index 00000000..e678b5a9 --- /dev/null +++ b/cometbft/v0.38/docs/core/Running-in-production.mdx @@ -0,0 +1,413 @@ +--- +order: 4 +--- + +# Running in production + +## Database + +By default, CometBFT uses the `syndtr/goleveldb` package for its in-process +key-value database. If you want maximal performance, it may be best to install +the real C-implementation of LevelDB and compile CometBFT to use that using +`make build COMETBFT_BUILD_OPTIONS=cleveldb`. See the [install +instructions](../guides/install.md) for details. + +CometBFT keeps multiple distinct databases in the `$CMTHOME/data`: + +- `blockstore.db`: Keeps the entire blockchain - stores blocks, + block commits, and block metadata, each indexed by height. Used to sync new + peers. +- `evidence.db`: Stores all verified evidence of misbehavior. +- `state.db`: Stores the current blockchain state (i.e. height, validators, + consensus params). Only grows if consensus params or validators change. Also + used to temporarily store intermediate results during block processing. +- `tx_index.db`: Indexes transactions and by tx hash and height. The tx results are indexed if they are added to the `FinalizeBlock` response in the application. + +By default, CometBFT will only index transactions by their hash and height, if +you want the result events to be indexed, see [indexing +transactions](../app-dev/indexing-transactions.md) for for details. + +Applications can expose block pruning strategies to the node operator. +Please read the documentation of your application to find out more details. + +Applications can use [state sync](./state-sync.md) to help nodes bootstrap quickly. + +## Logging + +Default logging level (`log_level = "main:info,state:info,statesync:info,*:error"`) should suffice for +normal operation mode. Read [this +post](https://blog.cosmos.network/one-of-the-exciting-new-features-in-0-10-0-release-is-smart-log-level-flag-e2506b4ab756) +for details on how to configure `log_level` config variable. Some of the +modules can be found [here](./how-to-read-logs.md#list-of-modules). If +you're trying to debug CometBFT or asked to provide logs with debug +logging level, you can do so by running CometBFT with +`--log_level="*:debug"`. + +## Write Ahead Logs (WAL) + +CometBFT uses write ahead logs for the consensus (`cs.wal`) and the mempool +(`mempool.wal`). Both WALs have a max size of 1GB and are automatically rotated. + +### Consensus WAL + +The `consensus.wal` is used to ensure we can recover from a crash at any point +in the consensus state machine. +It writes all consensus messages (timeouts, proposals, block part, or vote) +to a single file, flushing to disk before processing messages from its own +validator. Since CometBFT validators are expected to never sign a conflicting vote, the +WAL ensures we can always recover deterministically to the latest state of the consensus without +using the network or re-signing any consensus messages. + +If your `consensus.wal` is corrupted, see [below](#wal-corruption). + +### Mempool WAL + +The `mempool.wal` logs all incoming transactions before running CheckTx, but is +otherwise not used in any programmatic way. It's just a kind of manual +safe guard. Note the mempool provides no durability guarantees - a tx sent to one or many nodes +may never make it into the blockchain if those nodes crash before being able to +propose it. Clients must monitor their transactions by subscribing over websockets, +polling for them, or using `/broadcast_tx_commit`. In the worst case, transactions can be +resent from the mempool WAL manually. + +For the above reasons, the `mempool.wal` is disabled by default. To enable, set +`mempool.wal_dir` to where you want the WAL to be located (e.g. +`data/mempool.wal`). + +## DoS Exposure and Mitigation + +Validators are supposed to setup [Sentry Node Architecture](./validators.md) +to prevent Denial-of-Service attacks. + +### P2P + +The core of the CometBFT peer-to-peer system is `MConnection`. Each +connection has `MaxPacketMsgPayloadSize`, which is the maximum packet +size and bounded send & receive queues. One can impose restrictions on +send & receive rate per connection (`SendRate`, `RecvRate`). + +The number of open P2P connections can become quite large, and hit the operating system's open +file limit (since TCP connections are considered files on UNIX-based systems). Nodes should be +given a sizable open file limit, e.g. 8192, via `ulimit -n 8192` or other deployment-specific +mechanisms. + +### RPC + +#### Attack Exposure and Mitigation + +**It is generally not recommended for RPC endpoints to be exposed publicly, and +especially so if the node in question is a validator**, as the CometBFT RPC does +not currently provide advanced security features. Public exposure of RPC +endpoints without appropriate protection can make the associated node vulnerable +to a variety of attacks. + +It is entirely up to operators to ensure, if nodes' RPC endpoints have to be +exposed publicly, that appropriate measures have been taken to mitigate against +attacks. Some examples of mitigation measures include, but are not limited to: + +- Never publicly exposing the RPC endpoints of validators (i.e. if the RPC + endpoints absolutely have to be exposed, ensure you do so only on full nodes + and with appropriate protection) +- Correct usage of rate-limiting, authentication and caching (e.g. as provided + by reverse proxies like [nginx](https://nginx.org/) and/or DDoS protection + services like [Cloudflare](https://www.cloudflare.com)) +- Only exposing the specific endpoints absolutely necessary for the relevant use + cases (configurable via nginx/Cloudflare/etc.) + +If no expertise is available to the operator to assist with securing nodes' RPC +endpoints, it is strongly recommended to never expose those endpoints publicly. + +**Under no condition should any of the [unsafe RPC endpoints](../rpc/#/Unsafe) +ever be exposed publicly.** + +#### Endpoints Returning Multiple Entries + +Endpoints returning multiple entries are limited by default to return 30 +elements (100 max). See the [RPC Documentation](https://docs.cometbft.com/v0.38/rpc/) +for more information. + +## Debugging CometBFT + +If you ever have to debug CometBFT, the first thing you should probably do is +check out the logs. See [How to read logs](./how-to-read-logs.md), where we +explain what certain log statements mean. + +If, after skimming through the logs, things are not clear still, the next thing +to try is querying the `/status` RPC endpoint. It provides the necessary info: +whenever the node is syncing or not, what height it is on, etc. + +```bash +curl http(s)://{ip}:{rpcPort}/status +``` + +`/dump_consensus_state` will give you a detailed overview of the consensus +state (proposer, latest validators, peers states). From it, you should be able +to figure out why, for example, the network had halted. + +```bash +curl http(s)://{ip}:{rpcPort}/dump_consensus_state +``` + +There is a reduced version of this endpoint - `/consensus_state`, which returns +just the votes seen at the current height. + +If, after consulting with the logs and above endpoints, you still have no idea +what's happening, consider using `cometbft debug kill` sub-command. This +command will scrap all the available info and kill the process. See +[Debugging](../tools/debugging.md) for the exact format. + +You can inspect the resulting archive yourself or create an issue on +[Github](https://github.com/cometbft/cometbft). Before opening an issue +however, be sure to check if there's [no existing +issue](https://github.com/cometbft/cometbft/issues) already. + +## Monitoring CometBFT + +Each CometBFT instance has a standard `/health` RPC endpoint, which responds +with 200 (OK) if everything is fine and 500 (or no response) - if something is +wrong. + +Other useful endpoints include mentioned earlier `/status`, `/net_info` and +`/validators`. + +CometBFT also can report and serve Prometheus metrics. See +[Metrics](./metrics.md). + +`cometbft debug dump` sub-command can be used to periodically dump useful +information into an archive. See [Debugging](../tools/debugging.md) for more +information. + +## What happens when my app dies + +You are supposed to run CometBFT under a [process +supervisor](https://en.wikipedia.org/wiki/Process_supervision) (like +systemd or runit). It will ensure CometBFT is always running (despite +possible errors). + +Getting back to the original question, if your application dies, +CometBFT will panic. After a process supervisor restarts your +application, CometBFT should be able to reconnect successfully. The +order of restart does not matter for it. + +## Signal handling + +We catch SIGINT and SIGTERM and try to clean up nicely. For other +signals we use the default behavior in Go: +[Default behavior of signals in Go programs](https://golang.org/pkg/os/signal/#hdr-Default_behavior_of_signals_in_Go_programs). + +## Corruption + +**NOTE:** Make sure you have a backup of the CometBFT data directory. + +### Possible causes + +Remember that most corruption is caused by hardware issues: + +- RAID controllers with faulty / worn out battery backup, and an unexpected power loss +- Hard disk drives with write-back cache enabled, and an unexpected power loss +- Cheap SSDs with insufficient power-loss protection, and an unexpected power-loss +- Defective RAM +- Defective or overheating CPU(s) + +Other causes can be: + +- Database systems configured with fsync=off and an OS crash or power loss +- Filesystems configured to use write barriers plus a storage layer that ignores write barriers. LVM is a particular culprit. +- CometBFT bugs +- Operating system bugs +- Admin error (e.g., directly modifying CometBFT data-directory contents) + +(Source: [https://wiki.postgresql.org/wiki/Corruption](https://wiki.postgresql.org/wiki/Corruption)) + +### WAL Corruption + +If consensus WAL is corrupted at the latest height and you are trying to start +CometBFT, replay will fail with panic. + +Recovering from data corruption can be hard and time-consuming. Here are two approaches you can take: + +1. Delete the WAL file and restart CometBFT. It will attempt to sync with other peers. +2. Try to repair the WAL file manually: + +1) Create a backup of the corrupted WAL file: + + ```sh + cp "$CMTHOME/data/cs.wal/wal" > /tmp/corrupted_wal_backup + ``` + +2) Use `./scripts/wal2json` to create a human-readable version: + + ```sh + ./scripts/wal2json/wal2json "$CMTHOME/data/cs.wal/wal" > /tmp/corrupted_wal + ``` + +3) Search for a "CORRUPTED MESSAGE" line. +4) By looking at the previous message and the message after the corrupted one + and looking at the logs, try to rebuild the message. If the consequent + messages are marked as corrupted too (this may happen if length header + got corrupted or some writes did not make it to the WAL ~ truncation), + then remove all the lines starting from the corrupted one and restart + CometBFT. + + ```sh + $EDITOR /tmp/corrupted_wal + ``` + +5) After editing, convert this file back into binary form by running: + + ```sh + ./scripts/json2wal/json2wal /tmp/corrupted_wal $CMTHOME/data/cs.wal/wal + ``` + +## Hardware + +### Processor and Memory + +While actual specs vary depending on the load and validators count, minimal +requirements are: + +- 1GB RAM +- 25GB of disk space +- 1.4 GHz CPU + +SSD disks are preferable for applications with high transaction throughput. + +Recommended: + +- 2GB RAM +- 100GB SSD +- x64 2.0 GHz 2v CPU + +While for now, CometBFT stores all the history and it may require significant +disk space over time, we are planning to implement state syncing (See [this +issue](https://github.com/tendermint/tendermint/issues/828)). So, storing all +the past blocks will not be necessary. + +### Validator signing on 32 bit architectures (or ARM) + +Both our `ed25519` and `secp256k1` implementations require constant time +`uint64` multiplication. Non-constant time crypto can (and has) leaked +private keys on both `ed25519` and `secp256k1`. This doesn't exist in hardware +on 32 bit x86 platforms ([source](https://bearssl.org/ctmul.html)), and it +depends on the compiler to enforce that it is constant time. It's unclear at +this point whenever the Golang compiler does this correctly for all +implementations. + +**We do not support nor recommend running a validator on 32 bit architectures OR +the "VIA Nano 2000 Series", and the architectures in the ARM section rated +"S-".** + +### Operating Systems + +CometBFT can be compiled for a wide range of operating systems thanks to Go +language (the list of \$OS/\$ARCH pairs can be found +[here](https://golang.org/doc/install/source#environment)). + +While we do not favor any operation system, more secure and stable Linux server +distributions (like CentOS) should be preferred over desktop operation systems +(like Mac OS). + +### Miscellaneous + +NOTE: if you are going to use CometBFT in a public domain, make sure +you read [hardware recommendations](https://cosmos.network/validators) for a validator in the +Cosmos network. + +## Configuration parameters + +- `p2p.flush_throttle_timeout` +- `p2p.max_packet_msg_payload_size` +- `p2p.send_rate` +- `p2p.recv_rate` + +If you are going to use CometBFT in a private domain and you have a +private high-speed network among your peers, it makes sense to lower +flush throttle timeout and increase other params. + +```toml +[p2p] + +send_rate=20000000 # 2MB/s +recv_rate=20000000 # 2MB/s +flush_throttle_timeout=10 +max_packet_msg_payload_size=10240 # 10KB +``` + +- `mempool.recheck` + +After every block, CometBFT rechecks every transaction left in the +mempool to see if transactions committed in that block affected the +application state, so some of the transactions left may become invalid. +If that does not apply to your application, you can disable it by +setting `mempool.recheck=false`. + +- `mempool.broadcast` + +Setting this to false will stop the mempool from relaying transactions +to other peers until they are included in a block. It means only the +peer you send the tx to will see it until it is included in a block. + +- `consensus.skip_timeout_commit` + +We want `skip_timeout_commit=false` when there is economics on the line +because proposers should wait to hear for more votes. But if you don't +care about that and want the fastest consensus, you can skip it. It will +be kept false by default for public deployments (e.g. [Cosmos +Hub](https://hub.cosmos.network/)) while for enterprise +applications, setting it to true is not a problem. + +- `consensus.peer_gossip_sleep_duration` + +You can try to reduce the time your node sleeps before checking if +theres something to send its peers. + +- `consensus.timeout_commit` + +You can also try lowering `timeout_commit` (time we sleep before +proposing the next block). + +- `p2p.addr_book_strict` + +By default, CometBFT checks whenever a peer's address is routable before +saving it to the address book. The address is considered as routable if the IP +is [valid and within allowed ranges](https://github.com/cometbft/cometbft/blob/v0.38.x/p2p/netaddress.go#L258). + +This may not be the case for private or local networks, where your IP range is usually +strictly limited and private. If that case, you need to set `addr_book_strict` +to `false` (turn it off). + +- `rpc.max_open_connections` + +By default, the number of simultaneous connections is limited because most OS +give you limited number of file descriptors. + +If you want to accept greater number of connections, you will need to increase +these limits. + +[Sysctls to tune the system to be able to open more connections](https://github.com/satori-com/tcpkali/blob/master/doc/tcpkali.man.md#sysctls-to-tune-the-system-to-be-able-to-open-more-connections) + +The process file limits must also be increased, e.g. via `ulimit -n 8192`. + +...for N connections, such as 50k: + +```md +kern.maxfiles=10000+2*N # BSD +kern.maxfilesperproc=100+2*N # BSD +kern.ipc.maxsockets=10000+2*N # BSD +fs.file-max=10000+2*N # Linux +net.ipv4.tcp_max_orphans=N # Linux + +# For load-generating clients. +net.ipv4.ip_local_port_range="10000 65535" # Linux. +net.inet.ip.portrange.first=10000 # BSD/Mac. +net.inet.ip.portrange.last=65535 # (Enough for N < 55535) +net.ipv4.tcp_tw_reuse=1 # Linux +net.inet.tcp.maxtcptw=2*N # BSD + +# If using netfilter on Linux: +net.netfilter.nf_conntrack_max=N +echo $((N/8)) > /sys/module/nf_conntrack/parameters/hashsize +``` + +The similar option exists for limiting the number of gRPC connections - +`rpc.grpc_max_open_connections`. diff --git a/cometbft/v0.38/docs/core/Subscribing-to-events-via-Websocket.mdx b/cometbft/v0.38/docs/core/Subscribing-to-events-via-Websocket.mdx new file mode 100644 index 00000000..7b7f1512 --- /dev/null +++ b/cometbft/v0.38/docs/core/Subscribing-to-events-via-Websocket.mdx @@ -0,0 +1,96 @@ +--- +order: 7 +--- + +# Subscribing to events via Websocket + +CometBFT emits different events, which you can subscribe to via +[Websocket](https://en.wikipedia.org/wiki/WebSocket). This can be useful +for third-party applications (for analysis) or for inspecting state. + +[List of events](https://godoc.org/github.com/cometbft/cometbft/types#pkg-constants) + +To connect to a node via websocket from the CLI, you can use a tool such as +[wscat](https://github.com/websockets/wscat) and run: + +```sh +wscat -c ws://127.0.0.1:26657/websocket +``` + +NOTE: If your node's RPC endpoint is TLS-enabled, utilize the scheme `wss` instead of `ws`. + +You can subscribe to any of the events above by calling the `subscribe` RPC +method via Websocket along with a valid query. + +```json +{ + "jsonrpc": "2.0", + "method": "subscribe", + "id": 0, + "params": { + "query": "tm.event='NewBlock'" + } +} +``` + +Check out [API docs](https://docs.cometbft.com/v0.38/rpc/) for +more information on query syntax and other options. + +You can also use tags, given you had included them into FinalizeBlock +response, to query transaction results. See [Indexing +transactions](../app-dev/indexing-transactions.md) for details. + +## Query parameter and event type restrictions + +While CometBFT imposes no restrictions on the application with regards to the type of +the event output, there are several considerations that need to be taken into account +when querying events with numeric values. + +- Queries convert all numeric event values to `big.Float` , provided by `math/big`. Integers +are converted into a float with a precision equal to the number of bits needed +to represent this integer. This is done to avoid precision loss for big integers when they +are converted with the default precision (`64`). +- When comparing two values, if either one of them is a float, the other one will be represented +as a big float. Integers are again parsed as big floats with a precision equal to the number +of bits required to represent them. +- As with all floating point comparisons, comparing floats with decimal values can lead to imprecise +results. +- Queries cannot include negative numbers + +Prior to version `v0.38.x`, floats were not supported as query parameters. + +## ValidatorSetUpdates + +When validator set changes, ValidatorSetUpdates event is published. The +event carries a list of pubkey/power pairs. The list is the same +CometBFT receives from ABCI application (see [EndBlock +section](https://github.com/cometbft/cometbft/blob/v0.38.x/spec/abci/abci++_methods.md#endblock) in +the ABCI spec). + +Response: + +```json +{ + "jsonrpc": "2.0", + "id": 0, + "result": { + "query": "tm.event='ValidatorSetUpdates'", + "data": { + "type": "tendermint/event/ValidatorSetUpdates", + "value": { + "validator_updates": [ + { + "address": "09EAD022FD25DE3A02E64B0FE9610B1417183EE4", + "pub_key": { + "type": "tendermint/PubKeyEd25519", + "value": "ww0z4WaZ0Xg+YI10w43wTWbBmM3dpVza4mmSQYsd0ck=" + }, + "voting_power": "10", + "proposer_priority": "0" + } + ] + } + } + } +} +``` diff --git a/cometbft/v0.38/docs/core/Using-CometBFT.mdx b/cometbft/v0.38/docs/core/Using-CometBFT.mdx new file mode 100644 index 00000000..9918f2d8 --- /dev/null +++ b/cometbft/v0.38/docs/core/Using-CometBFT.mdx @@ -0,0 +1,572 @@ +--- +order: 2 +--- +{/* trigger rebuild */} + +# Using CometBFT + +This is a guide to using the `cometbft` program from the command line. +It assumes only that you have the `cometbft` binary installed and have +some rudimentary idea of what CometBFT and ABCI are. + +You can see the help menu with `cometbft --help`, and the version +number with `cometbft version`. + +## Directory Root + +The default directory for blockchain data is `~/.cometbft`. Override +this by setting the `CMTHOME` environment variable. + +## Initialize + +Initialize the root directory by running: + +```sh +cometbft init +``` + +This will create a new private key (`priv_validator_key.json`), and a +genesis file (`genesis.json`) containing the associated public key, in +`$CMTHOME/config`. This is all that's necessary to run a local testnet +with one validator. + +For more elaborate initialization, see the testnet command: + +```sh +cometbft testnet --help +``` + +### Genesis + +The `genesis.json` file in `$CMTHOME/config/` defines the initial +CometBFT state upon genesis of the blockchain ([see +definition](https://github.com/cometbft/cometbft/blob/v0.38.x/types/genesis.go)). + +#### Fields + +- `genesis_time`: Official time of blockchain start. +- `chain_id`: ID of the blockchain. **This must be unique for + every blockchain.** If your testnet blockchains do not have unique + chain IDs, you will have a bad time. The ChainID must be less than 50 symbols. +- `initial_height`: Height at which CometBFT should begin at. If a blockchain is conducting a network upgrade, + starting from the stopped height brings uniqueness to previous heights. +- `consensus_params` ([see spec](https://github.com/cometbft/cometbft/blob/v0.38.x/spec/core/data_structures.md#consensusparams)) + - `block` + - `max_bytes`: Max block size, in bytes. + - `max_gas`: Max gas per block. + - `evidence` + - `max_age_num_blocks`: Max age of evidence, in blocks. The basic formula + for calculating this is: MaxAgeDuration / (average block time). + - `max_age_duration`: Max age of evidence, in time. It should correspond + with an app's "unbonding period" or other similar mechanism for handling + [Nothing-At-Stake + attacks](https://vitalik.ca/general/2017/12/31/pos_faq.html#what-is-the-nothing-at-stake-problem-and-how-can-it-be-fixed). + - `max_bytes`: This sets the maximum size in bytes of evidence that can be committed + in a single block and should fall comfortably under the max block bytes. + - `validator` + - `pub_key_types`: Public key types validators can use. + - `version` + - `app_version`: ABCI application version. +- `validators`: List of initial validators. Note this may be overridden entirely by the + application, and may be left empty to make explicit that the + application will initialize the validator set upon `InitChain`. + - `pub_key`: The first element specifies the key type, + using the declared `PubKeyName` for the adopted + [key type](https://github.com/cometbft/cometbft/blob/v0.38.x/crypto/ed25519/ed25519.go#L36). + The second element are the pubkey bytes. + - `power`: The validator's voting power. + - `name`: Name of the validator (optional). +- `app_hash`: The expected application hash (as returned by the + `ResponseInfo` ABCI message) upon genesis. If the app's hash does + not match, CometBFT will panic. +- `app_state`: The application state (e.g. initial distribution + of tokens). + +> :warning: **ChainID must be unique to every blockchain. Reusing old chainID can cause issues** + +#### Sample genesis.json + +```json +{ + "genesis_time": "2023-01-21T11:17:42.341227868Z", + "chain_id": "test-chain-ROp9KF", + "initial_height": "0", + "consensus_params": { + "block": { + "max_bytes": "22020096", + "max_gas": "-1", + }, + "evidence": { + "max_age_num_blocks": "100000", + "max_age_duration": "172800000000000", + "max_bytes": 51200, + }, + "validator": { + "pub_key_types": [ + "ed25519" + ] + } + }, + "validators": [ + { + "address": "B547AB87E79F75A4A3198C57A8C2FDAF8628CB47", + "pub_key": { + "type": "tendermint/PubKeyEd25519", + "value": "P/V6GHuZrb8rs/k1oBorxc6vyXMlnzhJmv7LmjELDys=" + }, + "power": "10", + "name": "" + } + ], + "app_hash": "" +} +``` + +## Run + +To run a CometBFT node, use: + +```bash +cometbft node +``` + +By default, CometBFT will try to connect to an ABCI application on +`tcp://127.0.0.1:26658`. If you have the `kvstore` ABCI app installed, run it in +another window. If you don't, kill CometBFT and run an in-process version of +the `kvstore` app: + +```bash +cometbft node --proxy_app=kvstore +``` + +After a few seconds, you should see blocks start streaming in. Note that blocks +are produced regularly, even if there are no transactions. See [No Empty +Blocks](#no-empty-blocks), below, to modify this setting. + +CometBFT supports in-process versions of the `counter`, `kvstore`, and `noop` +apps that ship as examples with `abci-cli`. It's easy to compile your app +in-process with CometBFT if it's written in Go. If your app is not written in +Go, run it in another process, and use the `--proxy_app` flag to specify the +address of the socket it is listening on, for instance: + +```bash +cometbft node --proxy_app=/var/run/abci.sock +``` + +You can find out what flags are supported by running `cometbft node --help`. + +## Transactions + +To send a transaction, use `curl` to make requests to the CometBFT RPC +server, for example: + +```sh +curl http://localhost:26657/broadcast_tx_commit?tx=\"abcd\" +``` + +We can see the chain's status at the `/status` end-point: + +```sh +curl http://localhost:26657/status | json_pp +``` + +and the `latest_app_hash` in particular: + +```sh +curl http://localhost:26657/status | json_pp | grep latest_app_hash +``` + +Visit `http://localhost:26657` in your browser to see the list of other +endpoints. Some take no arguments (like `/status`), while others specify +the argument name and use `_` as a placeholder. + + +> TIP: Find the RPC Documentation [here](https://docs.cometbft.com/v0.38/rpc/) + +### Formatting + +The following nuances when sending/formatting transactions should be +taken into account: + +With `GET`: + +To send a UTF8 string byte array, quote the value of the tx parameter: + +```sh +curl 'http://localhost:26657/broadcast_tx_commit?tx="hello"' +``` + +which sends a 5 byte transaction: "h e l l o" \[68 65 6c 6c 6f\]. + +Note the URL must be wrapped with single quotes, else bash will ignore +the double quotes. To avoid the single quotes, escape the double quotes: + +```sh +curl http://localhost:26657/broadcast_tx_commit?tx=\"hello\" +``` + +Using a special character: + +```sh +curl 'http://localhost:26657/broadcast_tx_commit?tx="€5"' +``` + +sends a 4 byte transaction: "€5" (UTF8) \[e2 82 ac 35\]. + +To send as raw hex, omit quotes AND prefix the hex string with `0x`: + +```sh +curl http://localhost:26657/broadcast_tx_commit?tx=0x01020304 +``` + +which sends a 4 byte transaction: \[01 02 03 04\]. + +With `POST` (using `json`), the raw hex must be `base64` encoded: + +```sh +curl --data-binary '{"jsonrpc":"2.0","id":"anything","method":"broadcast_tx_commit","params": {"tx": "AQIDBA=="}}' -H 'content-type:text/plain;' http://localhost:26657 +``` + +which sends the same 4 byte transaction: \[01 02 03 04\]. + +Note that raw hex cannot be used in `POST` transactions. + +## Reset + +> :warning: **UNSAFE** Only do this in development and only if you can +afford to lose all blockchain data! + + +To reset a blockchain, stop the node and run: + +```sh +cometbft unsafe_reset_all +``` + +This command will remove the data directory and reset private validator and +address book files. + +## Configuration + +CometBFT uses a `config.toml` for configuration. For details, see [the +config specification](./configuration.md). + +Notable options include the socket address of the application +(`proxy_app`), the listening address of the CometBFT peer +(`p2p.laddr`), and the listening address of the RPC server +(`rpc.laddr`). + +Some fields from the config file can be overwritten with flags. + +## No Empty Blocks + +While the default behavior of `cometbft` is still to create blocks +approximately once per second, it is possible to disable empty blocks or +set a block creation interval. In the former case, blocks will be +created when there are new transactions or when the AppHash changes. + +To configure CometBFT to not produce empty blocks unless there are +transactions or the app hash changes, run CometBFT with this +additional flag: + +```sh +cometbft node --consensus.create_empty_blocks=false +``` + +or set the configuration via the `config.toml` file: + +```toml +[consensus] +create_empty_blocks = false +``` + +Remember: because the default is to _create empty blocks_, avoiding +empty blocks requires the config option to be set to `false`. + +The block interval setting allows for a delay (in time.Duration format [ParseDuration](https://golang.org/pkg/time/#ParseDuration)) between the +creation of each new empty block. It can be set with this additional flag: + +```sh +--consensus.create_empty_blocks_interval="5s" +``` + +or set the configuration via the `config.toml` file: + +```toml +[consensus] +create_empty_blocks_interval = "5s" +``` + +With this setting, empty blocks will be produced every 5s if no block +has been produced otherwise, regardless of the value of +`create_empty_blocks`. + +## Broadcast API + +Earlier, we used the `broadcast_tx_commit` endpoint to send a +transaction. When a transaction is sent to a CometBFT node, it will +run via `CheckTx` against the application. If it passes `CheckTx`, it +will be included in the mempool, broadcasted to other peers, and +eventually included in a block. + +Since there are multiple phases to processing a transaction, we offer +multiple endpoints to broadcast a transaction: + +```md +/broadcast_tx_async +/broadcast_tx_sync +/broadcast_tx_commit +``` + +These correspond to no-processing, processing through the mempool, and +processing through a block, respectively. That is, `broadcast_tx_async`, +will return right away without waiting to hear if the transaction is +even valid, while `broadcast_tx_sync` will return with the result of +running the transaction through `CheckTx`. Using `broadcast_tx_commit` +will wait until the transaction is committed in a block or until some +timeout is reached, but will return right away if the transaction does +not pass `CheckTx`. The return value for `broadcast_tx_commit` includes +two fields, `check_tx` and `deliver_tx`, pertaining to the result of +running the transaction through those ABCI messages. + +The benefit of using `broadcast_tx_commit` is that the request returns +after the transaction is committed (i.e. included in a block), but that +can take on the order of a second. For a quick result, use +`broadcast_tx_sync`, but the transaction will not be committed until +later, and by that point its effect on the state may change. + +Note the mempool does not provide strong guarantees - just because a tx passed +CheckTx (ie. was accepted into the mempool), doesn't mean it will be committed, +as nodes with the tx in their mempool may crash before they get to propose. +For more information, see the [mempool +write-ahead-log](./running-in-production.md#mempool-wal) + +## CometBFT Networks + +When `cometbft init` is run, both a `genesis.json` and +`priv_validator_key.json` are created in `~/.cometbft/config`. The +`genesis.json` might look like: + +```json +{ + "validators" : [ + { + "pub_key" : { + "value" : "h3hk+QE8c6QLTySp8TcfzclJw/BG79ziGB/pIA+DfPE=", + "type" : "tendermint/PubKeyEd25519" + }, + "power" : 10, + "name" : "" + } + ], + "app_hash" : "", + "chain_id" : "test-chain-rDlYSN", + "genesis_time" : "0001-01-01T00:00:00Z" +} +``` + +And the `priv_validator_key.json`: + +```json +{ + "last_step" : 0, + "last_round" : "0", + "address" : "B788DEDE4F50AD8BC9462DE76741CCAFF87D51E2", + "pub_key" : { + "value" : "h3hk+QE8c6QLTySp8TcfzclJw/BG79ziGB/pIA+DfPE=", + "type" : "tendermint/PubKeyEd25519" + }, + "last_height" : "0", + "priv_key" : { + "value" : "JPivl82x+LfVkp8i3ztoTjY6c6GJ4pBxQexErOCyhwqHeGT5ATxzpAtPJKnxNx/NyUnD8Ebv3OIYH+kgD4N88Q==", + "type" : "tendermint/PrivKeyEd25519" + } +} +``` + +The `priv_validator_key.json` actually contains a private key, and should +thus be kept absolutely secret; for now we work with the plain text. +Note the `last_` fields, which are used to prevent us from signing +conflicting messages. + +Note also that the `pub_key` (the public key) in the +`priv_validator_key.json` is also present in the `genesis.json`. + +The genesis file contains the list of public keys which may participate +in the consensus, and their corresponding voting power. Greater than 2/3 +of the voting power must be active (i.e. the corresponding private keys +must be producing signatures) for the consensus to make progress. In our +case, the genesis file contains the public key of our +`priv_validator_key.json`, so a CometBFT node started with the default +root directory will be able to make progress. Voting power uses an int64 +but must be positive, thus the range is: 0 through 9223372036854775807. +Because of how the current proposer selection algorithm works, we do not +recommend having voting powers greater than 10\^12 (ie. 1 trillion). + +If we want to add more nodes to the network, we have two choices: we can +add a new validator node, who will also participate in the consensus by +proposing blocks and voting on them, or we can add a new non-validator +node, who will not participate directly, but will verify and keep up +with the consensus protocol. + +### Peers + +#### Seed + +A seed node is a node who relays the addresses of other peers which they know +of. These nodes constantly crawl the network to try to get more peers. The +addresses which the seed node relays get saved into a local address book. Once +these are in the address book, you will connect to those addresses directly. +Basically the seed nodes job is just to relay everyones addresses. You won't +connect to seed nodes once you have received enough addresses, so typically you +only need them on the first start. The seed node will immediately disconnect +from you after sending you some addresses. + +#### Persistent Peer + +Persistent peers are people you want to be constantly connected with. If you +disconnect you will try to connect directly back to them as opposed to using +another address from the address book. On restarts you will always try to +connect to these peers regardless of the size of your address book. + +All peers relay peers they know of by default. This is called the peer exchange +protocol (PEX). With PEX, peers will be gossiping about known peers and forming +a network, storing peer addresses in the addrbook. Because of this, you don't +have to use a seed node if you have a live persistent peer. + +#### Connecting to Peers + +To connect to peers on start-up, specify them in the +`$CMTHOME/config/config.toml` or on the command line. Use `seeds` to +specify seed nodes, and +`persistent_peers` to specify peers that your node will maintain +persistent connections with. + +For example, + +```sh +cometbft node --p2p.seeds "f9baeaa15fedf5e1ef7448dd60f46c01f1a9e9c4@1.2.3.4:26656,0491d373a8e0fcf1023aaf18c51d6a1d0d4f31bd@5.6.7.8:26656" +``` + +Alternatively, you can use the `/dial_seeds` endpoint of the RPC to +specify seeds for a running node to connect to: + +```sh +curl 'localhost:26657/dial_seeds?seeds=\["f9baeaa15fedf5e1ef7448dd60f46c01f1a9e9c4@1.2.3.4:26656","0491d373a8e0fcf1023aaf18c51d6a1d0d4f31bd@5.6.7.8:26656"\]' +``` + +Note, with PEX enabled, you +should not need seeds after the first start. + +If you want CometBFT to connect to specific set of addresses and +maintain a persistent connection with each, you can use the +`--p2p.persistent_peers` flag or the corresponding setting in the +`config.toml` or the `/dial_peers` RPC endpoint to do it without +stopping CometBFT instance. + +```sh +cometbft node --p2p.persistent_peers "429fcf25974313b95673f58d77eacdd434402665@10.11.12.13:26656,96663a3dd0d7b9d17d4c8211b191af259621c693@10.11.12.14:26656" + +curl 'localhost:26657/dial_peers?persistent=true&peers=\["429fcf25974313b95673f58d77eacdd434402665@10.11.12.13:26656","96663a3dd0d7b9d17d4c8211b191af259621c693@10.11.12.14:26656"\]' +``` + +### Adding a Non-Validator + +Adding a non-validator is simple. Just copy the original `genesis.json` +to `~/.cometbft/config` on the new machine and start the node, +specifying seeds or persistent peers as necessary. If no seeds or +persistent peers are specified, the node won't make any blocks, because +it's not a validator, and it won't hear about any blocks, because it's +not connected to the other peer. + +### Adding a Validator + +The easiest way to add new validators is to do it in the `genesis.json`, +before starting the network. For instance, we could make a new +`priv_validator_key.json`, and copy it's `pub_key` into the above genesis. + +We can generate a new `priv_validator_key.json` with the command: + +```sh +cometbft gen_validator +``` + +Now we can update our genesis file. For instance, if the new +`priv_validator_key.json` looks like: + +```json +{ + "address" : "5AF49D2A2D4F5AD4C7C8C4CC2FB020131E9C4902", + "pub_key" : { + "value" : "l9X9+fjkeBzDfPGbUM7AMIRE6uJN78zN5+lk5OYotek=", + "type" : "tendermint/PubKeyEd25519" + }, + "priv_key" : { + "value" : "EDJY9W6zlAw+su6ITgTKg2nTZcHAH1NMTW5iwlgmNDuX1f35+OR4HMN88ZtQzsAwhETq4k3vzM3n6WTk5ii16Q==", + "type" : "tendermint/PrivKeyEd25519" + }, + "last_step" : 0, + "last_round" : "0", + "last_height" : "0" +} +``` + +then the new `genesis.json` will be: + +```json +{ + "validators" : [ + { + "pub_key" : { + "value" : "h3hk+QE8c6QLTySp8TcfzclJw/BG79ziGB/pIA+DfPE=", + "type" : "tendermint/PubKeyEd25519" + }, + "power" : 10, + "name" : "" + }, + { + "pub_key" : { + "value" : "l9X9+fjkeBzDfPGbUM7AMIRE6uJN78zN5+lk5OYotek=", + "type" : "cometbft/PubKeyEd25519" + }, + "power" : 10, + "name" : "" + } + ], + "app_hash" : "", + "chain_id" : "test-chain-rDlYSN", + "genesis_time" : "0001-01-01T00:00:00Z" +} +``` + +Update the `genesis.json` in `~/.cometbft/config`. Copy the genesis +file and the new `priv_validator_key.json` to the `~/.cometbft/config` on +a new machine. + +Now run `cometbft node` on both machines, and use either +`--p2p.persistent_peers` or the `/dial_peers` to get them to peer up. +They should start making blocks, and will only continue to do so as long +as both of them are online. + +To make a CometBFT network that can tolerate one of the validators +failing, you need at least four validator nodes (e.g., 2/3). + +Updating validators in a live network is supported but must be +explicitly programmed by the application developer. See the [application +developers guide](../app-dev/abci-cli.md) for more details. + +### Local Network + +To run a network locally, say on a single machine, you must change the `_laddr` +fields in the `config.toml` (or using the flags) so that the listening +addresses of the various sockets don't conflict. Additionally, you must set +`addr_book_strict=false` in the `config.toml`, otherwise CometBFT's p2p +library will deny making connections to peers with the same IP address. + +### Upgrading + +See the +[UPGRADING.md](https://github.com/cometbft/cometbft/blob/v0.38.x/UPGRADING.md) +guide. You may need to reset your chain between major breaking releases. +Although, we expect CometBFT to have fewer breaking releases in the future +(especially after 1.0 release). diff --git a/cometbft/v0.38/docs/core/Validators.mdx b/cometbft/v0.38/docs/core/Validators.mdx new file mode 100644 index 00000000..3d1b83a1 --- /dev/null +++ b/cometbft/v0.38/docs/core/Validators.mdx @@ -0,0 +1,101 @@ +--- +order: 6 +--- + +# Validators + +Validators are responsible for committing new blocks in the blockchain. +These validators participate in the consensus protocol by broadcasting +_votes_ which contain cryptographic signatures signed by each +validator's private key. + +Some Proof-of-Stake consensus algorithms aim to create a "completely" +decentralized system where all stakeholders (even those who are not always +available online) participate in the committing of blocks. CometBFT has a +different approach to block creation. Validators are expected to be online, and +the set of validators is permissioned/curated by the ABCI application. +Proof-of-stake is not required, but can be implemented on top of CometBFT +consensus. That is, validators may be required to post collateral on-chain, +off-chain, or may not be required to post any collateral at all. + +Validators have a cryptographic key-pair and an associated amount of +"voting power". Voting power need not be the same. + +## Becoming a Validator + +There are two ways to become validator. + +1. They can be pre-established in the [genesis state](./using-cometbft.md#genesis) +2. The ABCI app responds to the FinalizeBlock message with changes to the + existing validator set. + +## Setting up a Validator + +When setting up a validator there are countless ways to configure your setup. This guide is aimed at showing one of them, the sentry node design. This design is mainly for DDoS prevention. + +### Network Layout + +![ALT Network Layout](../imgs/sentry_layout.png) + +The diagram is based on AWS, other cloud providers will have similar solutions to design a solution. Running nodes is not limited to cloud providers, you can run nodes on bare metal systems as well. The architecture will be the same no matter which setup you decide to go with. + +The proposed network diagram is similar to the classical backend/frontend separation of services in a corporate environment. The “backend” in this case is the private network of the validator in the data center. The data center network might involve multiple subnets, firewalls and redundancy devices, which is not detailed on this diagram. The important point is that the data center allows direct connectivity to the chosen cloud environment. Amazon AWS has “Direct Connect”, while Google Cloud has “Partner Interconnect”. This is a dedicated connection to the cloud provider (usually directly to your virtual private cloud instance in one of the regions). + +All sentry nodes (the “frontend”) connect to the validator using this private connection. The validator does not have a public IP address to provide its services. + +Amazon has multiple availability zones within a region. One can install sentry nodes in other regions too. In this case the second, third and further regions need to have a private connection to the validator node. This can be achieved by VPC Peering (“VPC Network Peering” in Google Cloud). In this case, the second, third and further region sentry nodes will be directed to the first region and through the direct connect to the data center, arriving to the validator. + +A more persistent solution (not detailed on the diagram) is to have multiple direct connections to different regions from the data center. This way VPC Peering is not mandatory, although still beneficial for the sentry nodes. This overcomes the risk of depending on one region. It is more costly. + +### Local Configuration + +![ALT Local Configuration](../imgs/sentry_local_config.png) + +The validator will only talk to the sentry that are provided, the sentry nodes will communicate to the validator via a secret connection and the rest of the network through a normal connection. The sentry nodes do have the option of communicating with each other as well. + +When initializing nodes there are five parameters in the `config.toml` that may need to be altered. + +- `pex:` boolean. This turns the peer exchange reactor on or off for a node. When `pex=false`, only the `persistent_peers` list is available for connection. +- `persistent_peers:` a comma separated list of `nodeID@ip:port` values that define a list of peers that are expected to be online at all times. This is necessary at first startup because by setting `pex=false` the node will not be able to join the network. +- `unconditional_peer_ids:` comma separated list of nodeID's. These nodes will be connected to no matter the limits of inbound and outbound peers. This is useful for when sentry nodes have full address books. +- `private_peer_ids:` comma separated list of nodeID's. These nodes will not be gossiped to the network. This is an important field as you do not want your validator IP gossiped to the network. +- `addr_book_strict:` boolean. By default nodes with a routable address will be considered for connection. If this setting is turned off (false), non-routable IP addresses, like addresses in a private network can be added to the address book. +- `double_sign_check_height` int64 height. How many blocks to look back to check existence of the node's consensus votes before joining consensus When non-zero, the node will panic upon restart if the same consensus key was used to sign `double_sign_check_height` last blocks. So, validators should stop the state machine, wait for some blocks, and then restart the state machine to avoid panic. + +#### Validator Node Configuration + +| Config Option | Setting | +| ------------------------ | -------------------------- | +| pex | false | +| persistent_peers | list of sentry nodes | +| private_peer_ids | none | +| unconditional_peer_ids | optionally sentry node IDs | +| addr_book_strict | false | +| double_sign_check_height | 10 | + +The validator node should have `pex=false` so it does not gossip to the entire network. The persistent peers will be your sentry nodes. Private peers can be left empty as the validator is not trying to hide who it is communicating with. Setting unconditional peers is optional for a validator because they will not have a full address books. + +#### Sentry Node Configuration + +| Config Option | Setting | +| ---------------------- | --------------------------------------------- | +| pex | true | +| persistent_peers | validator node, optionally other sentry nodes | +| private_peer_ids | validator node ID | +| unconditional_peer_ids | validator node ID, optionally sentry node IDs | +| addr_book_strict | false | + +The sentry nodes should be able to talk to the entire network hence why `pex=true`. The persistent peers of a sentry node will be the validator, and optionally other sentry nodes. The sentry nodes should make sure that they do not gossip the validator's ip, to do this you must put the validators nodeID as a private peer. The unconditional peer IDs will be the validator ID and optionally other sentry nodes. + +> Note: Do not forget to secure your node's firewalls when setting them up. + +More Information can be found at these links: + +- [https://kb.certus.one/](https://kb.certus.one/) +- [https://forum.cosmos.network/t/sentry-node-architecture-overview/454](https://forum.cosmos.network/t/sentry-node-architecture-overview/454) + +### Validator keys + +Protecting a validator's consensus key is the most important factor to take in when designing your setup. The key that a validator is given upon creation of the node is called a consensus key, it has to be online at all times in order to vote on blocks. It is **not recommended** to merely hold your private key in the default json file (`priv_validator_key.json`). Fortunately, the [Interchain Foundation](https://interchain.io) has worked with a team to build a key management server for validators. You can find documentation on how to use it [here](https://github.com/iqlusioninc/tmkms), it is used extensively in production. You are not limited to using this tool, there are also [HSMs](https://safenet.gemalto.com/data-encryption/hardware-security-modules-hsms/), there is not a recommended HSM. + +Currently CometBFT uses [Ed25519](https://ed25519.cr.yp.to/) keys which are widely supported across the security sector and HSMs. diff --git a/cometbft/v0.38/docs/core/block-structure.mdx b/cometbft/v0.38/docs/core/block-structure.mdx new file mode 100644 index 00000000..e1e9175c --- /dev/null +++ b/cometbft/v0.38/docs/core/block-structure.mdx @@ -0,0 +1,19 @@ +--- +order: 8 +--- + +# Block Structure + +The CometBFT consensus engine records all agreements by a 2/3+ of nodes +into a blockchain, which is replicated among all nodes. This blockchain is +accessible via various RPC endpoints, mainly `/block?height=` to get the full +block, as well as `/blockchain?minHeight=_&maxHeight=_` to get a list of +headers. But what exactly is stored in these blocks? + +The [specification][data_structures] contains a detailed description of each +component - that's the best place to get started. + +To dig deeper, check out the [types package documentation][types]. + +[data_structures]: https://github.com/cometbft/cometbft/blob/v0.38.x/spec/core/data_structures.md +[types]: https://pkg.go.dev/github.com/cometbft/cometbft/types diff --git a/cometbft/v0.38/docs/core/block-sync.mdx b/cometbft/v0.38/docs/core/block-sync.mdx new file mode 100644 index 00000000..ad2797d4 --- /dev/null +++ b/cometbft/v0.38/docs/core/block-sync.mdx @@ -0,0 +1,49 @@ +--- +order: 10 +--- + +# Block Sync + +*Formerly known as Fast Sync* + +In a proof of work blockchain, syncing with the chain is the same +process as staying up-to-date with the consensus: download blocks, and +look for the one with the most total work. In proof-of-stake, the +consensus process is more complex, as it involves rounds of +communication between the nodes to determine what block should be +committed next. Using this process to sync up with the blockchain from +scratch can take a very long time. It's much faster to just download +blocks and check the merkle tree of validators than to run the real-time +consensus gossip protocol. + +## Using Block Sync + +When starting from scratch, nodes will use the Block Sync mode. +In this mode, the CometBFT daemon +will sync hundreds of times faster than if it used the real-time consensus +process. Once caught up, the daemon will switch out of Block Sync and into the +normal consensus mode. After running for some time, the node is considered +`caught up` if it has at least one peer and its height is at least as high as +the max reported peer height. See [the IsCaughtUp +method](https://github.com/cometbft/cometbft/blob/v0.38.x/blocksync/pool.go#L168). + +Note: While there have historically been multiple versions of blocksync, v0, v1, and v2, all versions +other than v0 have been deprecated in favor of the simplest and most well understood algorithm. + +```toml +####################################################### +### Block Sync Configuration Options ### +####################################################### +[blocksync] + +# Block Sync version to use: +# +# In v0.37, v1 and v2 of the block sync protocols were deprecated. +# Please use v0 instead. +# +# 1) "v0" - the default block sync implementation +version = "v0" +``` + +If we're lagging sufficiently, we should go back to block syncing, but +this is an [open issue](https://github.com/tendermint/tendermint/issues/129). diff --git a/cometbft/v0.38/docs/core/configuration.mdx b/cometbft/v0.38/docs/core/configuration.mdx new file mode 100644 index 00000000..8f5c6e04 --- /dev/null +++ b/cometbft/v0.38/docs/core/configuration.mdx @@ -0,0 +1,620 @@ +--- +order: 3 +--- + +# Configuration + +CometBFT can be configured via a TOML file in +`$CMTHOME/config/config.toml`. Some of these parameters can be overridden by +command-line flags. For most users, the options in the `##### main base configuration options #####` are intended to be modified while config options +further below are intended for advance power users. + +## Options + +The default configuration file create by `cometbft init` has all +the parameters set with their default values. It will look something +like the file below, however, double check by inspecting the +`config.toml` created with your version of `cometbft` installed: + +```toml +# This is a TOML config file. +# For more information, see https://github.com/toml-lang/toml + +# NOTE: Any path below can be absolute (e.g. "/var/myawesomeapp/data") or +# relative to the home directory (e.g. "data"). The home directory is +# "$HOME/.cometbft" by default, but could be changed via $CMTHOME env variable +# or --home cmd flag. + +# The version of the CometBFT binary that created or +# last modified the config file. Do not modify this. +version = "0.38.0" + +####################################################################### +### Main Base Config Options ### +####################################################################### + +# TCP or UNIX socket address of the ABCI application, +# or the name of an ABCI application compiled in with the CometBFT binary +proxy_app = "tcp://127.0.0.1:26658" + +# A custom human readable name for this node +moniker = "anonymous" + +# Database backend: goleveldb | cleveldb | boltdb | rocksdb | badgerdb +# * goleveldb (github.com/syndtr/goleveldb) +# - UNMAINTAINED +# - stable +# - pure go +# - stable +# * cleveldb (uses levigo wrapper) +# - fast +# - requires gcc +# - use cleveldb build tag (go build -tags cleveldb) +# * boltdb (uses etcd's fork of bolt - github.com/etcd-io/bbolt) +# - EXPERIMENTAL +# - may be faster is some use-cases (random reads - indexer) +# - use boltdb build tag (go build -tags boltdb) +# * rocksdb (uses github.com/tecbot/gorocksdb) +# - EXPERIMENTAL +# - requires gcc +# - use rocksdb build tag (go build -tags rocksdb) +# * badgerdb (uses github.com/dgraph-io/badger) +# - EXPERIMENTAL +# - use badgerdb build tag (go build -tags badgerdb) +db_backend = "goleveldb" + +# Database directory +db_dir = "data" + +# Output level for logging, including package level options +log_level = "info" + +# Output format: 'plain' (colored text) or 'json' +log_format = "plain" + +##### additional base config options ##### + +# Path to the JSON file containing the initial validator set and other meta data +genesis_file = "config/genesis.json" + +# Path to the JSON file containing the private key to use as a validator in the consensus protocol +priv_validator_key_file = "config/priv_validator_key.json" + +# Path to the JSON file containing the last sign state of a validator +priv_validator_state_file = "data/priv_validator_state.json" + +# TCP or UNIX socket address for CometBFT to listen on for +# connections from an external PrivValidator process +priv_validator_laddr = "" + +# Path to the JSON file containing the private key to use for node authentication in the p2p protocol +node_key_file = "config/node_key.json" + +# Mechanism to connect to the ABCI application: socket | grpc +abci = "socket" + +# If true, query the ABCI app on connecting to a new peer +# so the app can decide if we should keep the connection or not +filter_peers = false + + +####################################################################### +### Advanced Configuration Options ### +####################################################################### + +####################################################### +### RPC Server Configuration Options ### +####################################################### +[rpc] + +# TCP or UNIX socket address for the RPC server to listen on +laddr = "tcp://127.0.0.1:26657" + +# A list of origins a cross-domain request can be executed from +# Default value '[]' disables cors support +# Use '["*"]' to allow any origin +cors_allowed_origins = [] + +# A list of methods the client is allowed to use with cross-domain requests +cors_allowed_methods = ["HEAD", "GET", "POST", ] + +# A list of non simple headers the client is allowed to use with cross-domain requests +cors_allowed_headers = ["Origin", "Accept", "Content-Type", "X-Requested-With", "X-Server-Time", ] + +# TCP or UNIX socket address for the gRPC server to listen on +# NOTE: This server only supports /broadcast_tx_commit +grpc_laddr = "" + +# Maximum number of simultaneous connections. +# Does not include RPC (HTTP&WebSocket) connections. See max_open_connections +# If you want to accept a larger number than the default, make sure +# you increase your OS limits. +# 0 - unlimited. +# Should be < {ulimit -Sn} - {MaxNumInboundPeers} - {MaxNumOutboundPeers} - {N of wal, db and other open files} +# 1024 - 40 - 10 - 50 = 924 = ~900 +grpc_max_open_connections = 900 + +# Activate unsafe RPC commands like /dial_seeds and /unsafe_flush_mempool +unsafe = false + +# Maximum number of simultaneous connections (including WebSocket). +# Does not include gRPC connections. See grpc_max_open_connections +# If you want to accept a larger number than the default, make sure +# you increase your OS limits. +# 0 - unlimited. +# Should be < {ulimit -Sn} - {MaxNumInboundPeers} - {MaxNumOutboundPeers} - {N of wal, db and other open files} +# 1024 - 40 - 10 - 50 = 924 = ~900 +max_open_connections = 900 + +# Maximum number of unique clientIDs that can /subscribe +# If you're using /broadcast_tx_commit, set to the estimated maximum number +# of broadcast_tx_commit calls per block. +max_subscription_clients = 100 + +# Maximum number of unique queries a given client can /subscribe to +# If you're using GRPC (or Local RPC client) and /broadcast_tx_commit, set to +# the estimated # maximum number of broadcast_tx_commit calls per block. +max_subscriptions_per_client = 5 + +# Experimental parameter to specify the maximum number of events a node will +# buffer, per subscription, before returning an error and closing the +# subscription. Must be set to at least 100, but higher values will accommodate +# higher event throughput rates (and will use more memory). +experimental_subscription_buffer_size = 200 + +# Experimental parameter to specify the maximum number of RPC responses that +# can be buffered per WebSocket client. If clients cannot read from the +# WebSocket endpoint fast enough, they will be disconnected, so increasing this +# parameter may reduce the chances of them being disconnected (but will cause +# the node to use more memory). +# +# Must be at least the same as "experimental_subscription_buffer_size", +# otherwise connections could be dropped unnecessarily. This value should +# ideally be somewhat higher than "experimental_subscription_buffer_size" to +# accommodate non-subscription-related RPC responses. +experimental_websocket_write_buffer_size = 200 + +# If a WebSocket client cannot read fast enough, at present we may +# silently drop events instead of generating an error or disconnecting the +# client. +# +# Enabling this experimental parameter will cause the WebSocket connection to +# be closed instead if it cannot read fast enough, allowing for greater +# predictability in subscription behavior. +experimental_close_on_slow_client = false + +# How long to wait for a tx to be committed during /broadcast_tx_commit. +# WARNING: Using a value larger than 10s will result in increasing the +# global HTTP write timeout, which applies to all connections and endpoints. +# See https://github.com/tendermint/tendermint/issues/3435 +timeout_broadcast_tx_commit = "10s" + +# Maximum number of requests that can be sent in a JSON-RPC batch request. +# Possible values: number greater than 0. +# If the number of requests sent in a JSON-RPC batch exceed the maximum batch +# size configured, an error will be returned. +# The default value is set to `10`, which will limit the number of requests +# to 10 requests per a JSON-RPC batch request. +# If you don't want to enforce a maximum number of requests for a batch +# request set this value to `0`. +max_request_batch_size = 10 + +# Maximum size of request body, in bytes +max_body_bytes = 1000000 + +# Maximum size of request header, in bytes +max_header_bytes = 1048576 + +# The path to a file containing certificate that is used to create the HTTPS server. +# Might be either absolute path or path related to CometBFT's config directory. +# If the certificate is signed by a certificate authority, +# the certFile should be the concatenation of the server's certificate, any intermediates, +# and the CA's certificate. +# NOTE: both tls_cert_file and tls_key_file must be present for CometBFT to create HTTPS server. +# Otherwise, HTTP server is run. +tls_cert_file = "" + +# The path to a file containing matching private key that is used to create the HTTPS server. +# Might be either absolute path or path related to CometBFT's config directory. +# NOTE: both tls-cert-file and tls-key-file must be present for CometBFT to create HTTPS server. +# Otherwise, HTTP server is run. +tls_key_file = "" + +# pprof listen address (https://golang.org/pkg/net/http/pprof) +pprof_laddr = "" + +####################################################### +### P2P Configuration Options ### +####################################################### +[p2p] + +# Address to listen for incoming connections +laddr = "tcp://0.0.0.0:26656" + +# Address to advertise to peers for them to dial. If empty, will use the same +# port as the laddr, and will introspect on the listener to figure out the +# address. IP and port are required. Example: 159.89.10.97:26656 +external_address = "" + +# Comma separated list of seed nodes to connect to +seeds = "" + +# Comma separated list of nodes to keep persistent connections to +persistent_peers = "" + +# Path to address book +addr_book_file = "config/addrbook.json" + +# Set true for strict address routability rules +# Set false for private or local networks +addr_book_strict = true + +# Maximum number of inbound peers +max_num_inbound_peers = 40 + +# Maximum number of outbound peers to connect to, excluding persistent peers +max_num_outbound_peers = 10 + +# List of node IDs, to which a connection will be (re)established ignoring any existing limits +unconditional_peer_ids = "" + +# Maximum pause when redialing a persistent peer (if zero, exponential backoff is used) +persistent_peers_max_dial_period = "0s" + +# Time to wait before flushing messages out on the connection +flush_throttle_timeout = "100ms" + +# Maximum size of a message packet payload, in bytes +max_packet_msg_payload_size = 1024 + +# Rate at which packets can be sent, in bytes/second +send_rate = 5120000 + +# Rate at which packets can be received, in bytes/second +recv_rate = 5120000 + +# Set true to enable the peer-exchange reactor +pex = true + +# Seed mode, in which node constantly crawls the network and looks for +# peers. If another node asks it for addresses, it responds and disconnects. +# +# Does not work if the peer-exchange reactor is disabled. +seed_mode = false + +# Comma separated list of peer IDs to keep private (will not be gossiped to other peers) +private_peer_ids = "" + +# Toggle to disable guard against peers connecting from the same ip. +allow_duplicate_ip = false + +# Peer connection configuration. +handshake_timeout = "20s" +dial_timeout = "3s" + +####################################################### +### Mempool Configuration Option ### +####################################################### +[mempool] + +# The type of mempool for this node to use. +# +# Possible types: +# - "flood" : concurrent linked list mempool with flooding gossip protocol +# (default) +# - "nop" : nop-mempool (short for no operation; the ABCI app is responsible +# for storing, disseminating and proposing txs). "create_empty_blocks=false" is +# not supported. +type = "flood" + +# Recheck (default: true) defines whether CometBFT should recheck the +# validity for all remaining transaction in the mempool after a block. +# Since a block affects the application state, some transactions in the +# mempool may become invalid. If this does not apply to your application, +# you can disable rechecking. +recheck = true + +# Broadcast (default: true) defines whether the mempool should relay +# transactions to other peers. Setting this to false will stop the mempool +# from relaying transactions to other peers until they are included in a +# block. In other words, if Broadcast is disabled, only the peer you send +# the tx to will see it until it is included in a block. +broadcast = true + +# WalPath (default: "") configures the location of the Write Ahead Log +# (WAL) for the mempool. The WAL is disabled by default. To enable, set +# wal_dir to where you want the WAL to be written (e.g. +# "data/mempool.wal"). +wal_dir = "" + +# Maximum number of transactions in the mempool +size = 5000 + +# Limit the total size of all txs in the mempool. +# This only accounts for raw transactions (e.g. given 1MB transactions and +# max_txs_bytes=5MB, mempool will only accept 5 transactions). +max_txs_bytes = 1073741824 + +# Size of the cache (used to filter transactions we saw earlier) in transactions +cache_size = 10000 + +# Do not remove invalid transactions from the cache (default: false) +# Set to true if it's not possible for any invalid transaction to become valid +# again in the future. +keep-invalid-txs-in-cache = false + +# Maximum size of a single transaction. +# NOTE: the max size of a tx transmitted over the network is {max_tx_bytes}. +max_tx_bytes = 1048576 + +# Maximum size of a batch of transactions to send to a peer +# Including space needed by encoding (one varint per transaction). +# XXX: Unused due to https://github.com/tendermint/tendermint/issues/5796 +max_batch_bytes = 0 + +####################################################### +### State Sync Configuration Options ### +####################################################### +[statesync] +# State sync rapidly bootstraps a new node by discovering, fetching, and restoring a state machine +# snapshot from peers instead of fetching and replaying historical blocks. Requires some peers in +# the network to take and serve state machine snapshots. State sync is not attempted if the node +# has any local state (LastBlockHeight > 0). The node will have a truncated block history, +# starting from the height of the snapshot. +enable = false + +# RPC servers (comma-separated) for light client verification of the synced state machine and +# retrieval of state data for node bootstrapping. Also needs a trusted height and corresponding +# header hash obtained from a trusted source, and a period during which validators can be trusted. +# +# For Cosmos SDK-based chains, trust_period should usually be about 2/3 of the unbonding time (~2 +# weeks) during which they can be financially punished (slashed) for misbehavior. +rpc_servers = "" +trust_height = 0 +trust_hash = "" +trust_period = "168h0m0s" + +# Time to spend discovering snapshots before initiating a restore. +discovery_time = "15s" + +# Temporary directory for state sync snapshot chunks, defaults to the OS tempdir (typically /tmp). +# Will create a new, randomly named directory within, and remove it when done. +temp_dir = "" + +# The timeout duration before re-requesting a chunk, possibly from a different +# peer (default: 1 minute). +chunk_request_timeout = "10s" + +# The number of concurrent chunk fetchers to run (default: 1). +chunk_fetchers = "4" + +####################################################### +### Block Sync Configuration Options ### +####################################################### +[blocksync] + +# Block Sync version to use: +# +# In v0.37, v1 and v2 of the block sync protocols were deprecated. +# Please use v0 instead. +# +# 1) "v0" - the default block sync implementation +version = "v0" + +####################################################### +### Consensus Configuration Options ### +####################################################### +[consensus] + +wal_file = "data/cs.wal/wal" + +# How long we wait for a proposal block before prevoting nil +timeout_propose = "3s" +# How much timeout_propose increases with each round +timeout_propose_delta = "500ms" +# How long we wait after receiving +2/3 prevotes for “anything” (ie. not a single block or nil) +timeout_prevote = "1s" +# How much the timeout_prevote increases with each round +timeout_prevote_delta = "500ms" +# How long we wait after receiving +2/3 precommits for “anything” (ie. not a single block or nil) +timeout_precommit = "1s" +# How much the timeout_precommit increases with each round +timeout_precommit_delta = "500ms" +# How long we wait after committing a block, before starting on the new +# height (this gives us a chance to receive some more precommits, even +# though we already have +2/3). +timeout_commit = "1s" + +# How many blocks to look back to check existence of the node's consensus votes before joining consensus +# When non-zero, the node will panic upon restart +# if the same consensus key was used to sign {double_sign_check_height} last blocks. +# So, validators should stop the state machine, wait for some blocks, and then restart the state machine to avoid panic. +double_sign_check_height = 0 + +# Make progress as soon as we have all the precommits (as if TimeoutCommit = 0) +skip_timeout_commit = false + +# EmptyBlocks mode and possible interval between empty blocks +create_empty_blocks = true +create_empty_blocks_interval = "0s" + +# Reactor sleep duration parameters +peer_gossip_sleep_duration = "100ms" +peer_query_maj23_sleep_duration = "2s" + +####################################################### +### Storage Configuration Options ### +####################################################### +[storage] + +# Set to true to discard ABCI responses from the state store, which can save a +# considerable amount of disk space. Set to false to ensure ABCI responses are +# persisted. ABCI responses are required for /block_results RPC queries, and to +# reindex events in the command-line tool. +discard_abci_responses = false + +####################################################### +### Transaction Indexer Configuration Options ### +####################################################### +[tx_index] + +# What indexer to use for transactions +# +# The application will set which txs to index. In some cases a node operator will be able +# to decide which txs to index based on configuration set in the application. +# +# Options: +# 1) "null" +# 2) "kv" (default) - the simplest possible indexer, backed by key-value storage (defaults to levelDB; see DBBackend). +# - When "kv" is chosen "tx.height" and "tx.hash" will always be indexed. +# 3) "psql" - the indexer services backed by PostgreSQL. +# When "kv" or "psql" is chosen "tx.height" and "tx.hash" will always be indexed. +indexer = "kv" + +# The PostgreSQL connection configuration, the connection format: +# postgresql://:@:/? +psql-conn = "" + +####################################################### +### Instrumentation Configuration Options ### +####################################################### +[instrumentation] + +# When true, Prometheus metrics are served under /metrics on +# PrometheusListenAddr. +# Check out the documentation for the list of available metrics. +prometheus = false + +# Address to listen for Prometheus collector(s) connections +prometheus_listen_addr = ":26660" + +# Maximum number of simultaneous connections. +# If you want to accept a larger number than the default, make sure +# you increase your OS limits. +# 0 - unlimited. +max_open_connections = 3 + +# Instrumentation namespace +namespace = "cometbft" + + ``` + +## Empty blocks VS no empty blocks + +### create_empty_blocks = true + +If `create_empty_blocks` is set to `true` in your config, blocks will be created ~ every second (with default consensus parameters). You can regulate the delay between blocks by changing the `timeout_commit`. E.g. `timeout_commit = "10s"` should result in ~ 10 second blocks. + +### create_empty_blocks = false + +In this setting, blocks are created when transactions received. + +Note after the block H, CometBFT creates something we call a "proof block" (only if the application hash changed) H+1. The reason for this is to support proofs. If you have a transaction in block H that changes the state to X, the new application hash will only be included in block H+1. If after your transaction is committed, you want to get a light-client proof for the new state (X), you need the new block to be committed in order to do that because the new block has the new application hash for the state X. That's why we make a new (empty) block if the application hash changes. Otherwise, you won't be able to make a proof for the new state. + +Plus, if you set `create_empty_blocks_interval` to something other than the default (`0`), CometBFT will be creating empty blocks even in the absence of transactions every `create_empty_blocks_interval.` For instance, with `create_empty_blocks = false` and `create_empty_blocks_interval = "30s"`, CometBFT will only create blocks if there are transactions, or after waiting 30 seconds without receiving any transactions. + +## Consensus timeouts explained + +There's a variety of information about timeouts in [Running in +production](./running-in-production.md#configuration-parameters). +You can also find more detailed explanation in the paper describing +the Tendermint consensus algorithm, adopted by CometBFT: [The latest +gossip on BFT consensus](https://arxiv.org/abs/1807.04938). + +```toml +[consensus] +... + +timeout_propose = "3s" +timeout_propose_delta = "500ms" +timeout_prevote = "1s" +timeout_prevote_delta = "500ms" +timeout_precommit = "1s" +timeout_precommit_delta = "500ms" +timeout_commit = "1s" +``` + +Note that in a successful round, the only timeout that we absolutely wait no +matter what is `timeout_commit`. +Here's a brief summary of the timeouts: + +- `timeout_propose` = how long a validator should wait for a proposal block before prevoting nil +- `timeout_propose_delta` = how much `timeout_propose` increases with each round +- `timeout_prevote` = how long a validator should wait after receiving +2/3 prevotes for + anything (ie. not a single block or nil) +- `timeout_prevote_delta` = how much the `timeout_prevote` increases with each round +- `timeout_precommit` = how long a validator should wait after receiving +2/3 precommits for + anything (ie. not a single block or nil) +- `timeout_precommit_delta` = how much the `timeout_precommit` increases with each round +- `timeout_commit` = how long a validator should wait after committing a block, before starting + on the new height (this gives us a chance to receive some more precommits, + even though we already have +2/3) + +### The adverse effect of using inconsistent `timeout_propose` in a network + +Here's an interesting question. What happens if a particular validator sets a +very small `timeout_propose`, as compared to the rest of the network? + +Imagine there are only two validators in your network: Alice and Bob. Bob sets +`timeout_propose` to 0s. Alice uses the default value of 3s. Let's say they +both have an equal voting power. Given the proposer selection algorithm is a +weighted round-robin, you may expect Alice and Bob to take turns proposing +blocks, and the result like: + +``` +#1 block - Alice +#2 block - Bob +#3 block - Alice +#4 block - Bob +... +``` + +What happens in reality is, however, a little bit different: + +``` +#1 block - Bob +#2 block - Bob +#3 block - Bob +#4 block - Bob +``` + +That's because Bob doesn't wait for a proposal from Alice (prevotes `nil`). +This leaves Alice no chances to commit a block. Note that every block Bob +creates needs a vote from Alice to constitute 2/3+. Bob always gets one because +Alice has `timeout_propose` set to 3s. Alice never gets one because Bob has it +set to 0s. + +Imagine now there are ten geographically distributed validators. One of them +(Bob) sets `timeout_propose` to 0s. Others have it set to 3s. Now, Bob won't be +able to move with his own speed because it still needs 2/3 votes of the other +validators and it takes time to propagate those. I.e., the network moves with +the speed of time to accumulate 2/3+ of votes (prevotes & precommits), not with +the speed of the fastest proposer. + +> Isn't block production determined by voting power? + +If it were determined solely by voting power, it wouldn't be possible to ensure +liveness. Timeouts exist because the network can't rely on a single proposer +being available and must move on if such is not responding. + +> How can we address situations where someone arbitrarily adjusts their block +> production time to gain an advantage? + +The impact shown above is negligible in a decentralized network with enough +decentralization. + +### The adverse effect of using inconsistent `timeout_commit` in a network + +Let's look at the same scenario as before. There are ten geographically +distributed validators. One of them (Bob) sets `timeout_commit` to 0s. Others +have it set to 1s (the default value). Now, Bob will be the fastest producer +because he doesn't wait for additional precommits after creating a block. If +waiting for precommits (`timeout_commit`) is not incentivized, Bob will accrue +more rewards compared to the other 9 validators. + +This is because Bob has the advantage of broadcasting its proposal early (1 +second earlier than the others). But it also makes it possible for Bob to miss +a proposal from another validator and prevote `nil` due to him starting +`timeout_propose` earlier. I.e., if Bob's `timeout_commit` is too low comparing +to other validators, then he might miss some proposals and get slashed for +inactivity. diff --git a/cometbft/v0.38/docs/core/how-to-read-logs.mdx b/cometbft/v0.38/docs/core/how-to-read-logs.mdx new file mode 100644 index 00000000..2e54579a --- /dev/null +++ b/cometbft/v0.38/docs/core/how-to-read-logs.mdx @@ -0,0 +1,141 @@ +--- +order: 7 +--- + +# How to read logs + +## Walkabout example + +We first create three connections (mempool, consensus and query) to the +application (running `kvstore` locally in this case). + +```sh +I[10-04|13:54:27.364] Starting multiAppConn module=proxy impl=multiAppConn +I[10-04|13:54:27.366] Starting localClient module=abci-client connection=query impl=localClient +I[10-04|13:54:27.366] Starting localClient module=abci-client connection=mempool impl=localClient +I[10-04|13:54:27.367] Starting localClient module=abci-client connection=consensus impl=localClient +``` + +Then CometBFT and the application perform a handshake. + +```sh +I[10-04|13:54:27.367] ABCI Handshake module=consensus appHeight=90 appHash=E0FBAFBF6FCED8B9786DDFEB1A0D4FA2501BADAD +I[10-04|13:54:27.368] ABCI Replay Blocks module=consensus appHeight=90 storeHeight=90 stateHeight=90 +I[10-04|13:54:27.368] Completed ABCI Handshake - CometBFT and App are synced module=consensus appHeight=90 appHash=E0FBAFBF6FCED8B9786DDFEB1A0D4FA2501BADAD +``` + +After that, we start a few more things like the event switch and reactors. + +```sh +I[10-04|13:54:27.374] Starting EventSwitch module=types impl=EventSwitch +I[10-04|13:54:27.375] This node is a validator module=consensus +I[10-04|13:54:27.379] Starting Node module=main impl=Node +I[10-04|13:54:27.381] Local listener module=p2p ip=:: port=26656 +I[10-04|13:54:30.386] Starting DefaultListener module=p2p impl=Listener(@10.0.2.15:26656) +I[10-04|13:54:30.387] Starting P2P Switch module=p2p impl="P2P Switch" +I[10-04|13:54:30.387] Starting MempoolReactor module=mempool impl=MempoolReactor +I[10-04|13:54:30.387] Starting BlockchainReactor module=blockchain impl=BlockchainReactor +I[10-04|13:54:30.387] Starting ConsensusReactor module=consensus impl=ConsensusReactor +I[10-04|13:54:30.387] ConsensusReactor module=consensus fastSync=false +I[10-04|13:54:30.387] Starting ConsensusState module=consensus impl=ConsensusState +I[10-04|13:54:30.387] Starting WAL module=consensus wal=/home/vagrant/.cometbft/data/cs.wal/wal impl=WAL +I[10-04|13:54:30.388] Starting TimeoutTicker module=consensus impl=TimeoutTicker +``` + +Notice the second row where CometBFT reports that "This node is a +validator". It also could be just an observer (regular node). + +Next we replay all the messages from the WAL. + +```sh +I[10-04|13:54:30.390] Catchup by replaying consensus messages module=consensus height=91 +I[10-04|13:54:30.390] Replay: New Step module=consensus height=91 round=0 step=RoundStepNewHeight +I[10-04|13:54:30.390] Replay: Done module=consensus +``` + +"Started node" message signals that everything is ready for work. + +```sh +I[10-04|13:54:30.391] Starting RPC HTTP server on tcp socket 0.0.0.0:26657 module=rpc-server +I[10-04|13:54:30.392] Started node module=main nodeInfo="NodeInfo{id: DF22D7C92C91082324A1312F092AA1DA197FA598DBBFB6526E, moniker: anonymous, network: test-chain-3MNw2N [remote , listen 10.0.2.15:26656], version: 0.11.0-10f361fc ([wire_version=0.6.2 p2p_version=0.5.0 consensus_version=v1/0.2.2 rpc_version=0.7.0/3 tx_index=on rpc_addr=tcp://0.0.0.0:26657])}" +``` + +Next follows a standard block creation cycle, where we enter a new +round, propose a block, receive more than 2/3 of prevotes, then +precommits and finally have a chance to commit a block. For details, +please refer to [Byzantine Consensus Algorithm](https://github.com/cometbft/cometbft/blob/v0.38.x/spec/consensus/consensus.md). + +```sh +I[10-04|13:54:30.393] enterNewRound(91/0). Current: 91/0/RoundStepNewHeight module=consensus +I[10-04|13:54:30.393] enterPropose(91/0). Current: 91/0/RoundStepNewRound module=consensus +I[10-04|13:54:30.393] enterPropose: Our turn to propose module=consensus proposer=125B0E3C5512F5C2B0E1109E31885C4511570C42 privValidator="PrivValidator{125B0E3C5512F5C2B0E1109E31885C4511570C42 LH:90, LR:0, LS:3}" +I[10-04|13:54:30.394] Signed proposal module=consensus height=91 round=0 proposal="Proposal{91/0 1:21B79872514F (-1,:0:000000000000) {/10EDEDD7C84E.../}}" +I[10-04|13:54:30.397] Received complete proposal block module=consensus height=91 hash=F671D562C7B9242900A286E1882EE64E5556FE9E +I[10-04|13:54:30.397] enterPrevote(91/0). Current: 91/0/RoundStepPropose module=consensus +I[10-04|13:54:30.397] enterPrevote: ProposalBlock is valid module=consensus height=91 round=0 +I[10-04|13:54:30.398] Signed and pushed vote module=consensus height=91 round=0 vote="Vote{0:125B0E3C5512 91/00/1(Prevote) F671D562C7B9 {/89047FFC21D8.../}}" err=null +I[10-04|13:54:30.401] Added to prevote module=consensus vote="Vote{0:125B0E3C5512 91/00/1(Prevote) F671D562C7B9 {/89047FFC21D8.../}}" prevotes="VoteSet{H:91 R:0 T:1 +2/3:F671D562C7B9242900A286E1882EE64E5556FE9E:1:21B79872514F BA{1:X} map[]}" +I[10-04|13:54:30.401] enterPrecommit(91/0). Current: 91/0/RoundStepPrevote module=consensus +I[10-04|13:54:30.401] enterPrecommit: +2/3 prevoted proposal block. Locking module=consensus hash=F671D562C7B9242900A286E1882EE64E5556FE9E +I[10-04|13:54:30.402] Signed and pushed vote module=consensus height=91 round=0 vote="Vote{0:125B0E3C5512 91/00/2(Precommit) F671D562C7B9 {/80533478E41A.../}}" err=null +I[10-04|13:54:30.404] Added to precommit module=consensus vote="Vote{0:125B0E3C5512 91/00/2(Precommit) F671D562C7B9 {/80533478E41A.../}}" precommits="VoteSet{H:91 R:0 T:2 +2/3:F671D562C7B9242900A286E1882EE64E5556FE9E:1:21B79872514F BA{1:X} map[]}" +I[10-04|13:54:30.404] enterCommit(91/0). Current: 91/0/RoundStepPrecommit module=consensus +I[10-04|13:54:30.405] Finalizing commit of block with 0 txs module=consensus height=91 hash=F671D562C7B9242900A286E1882EE64E5556FE9E root=E0FBAFBF6FCED8B9786DDFEB1A0D4FA2501BADAD +I[10-04|13:54:30.405] Block{ + Header{ + ChainID: test-chain-3MNw2N + Height: 91 + Time: 2017-10-04 13:54:30.393 +0000 UTC + NumTxs: 0 + LastBlockID: F15AB8BEF9A6AAB07E457A6E16BC410546AA4DC6:1:D505DA273544 + LastCommit: 56FEF2EFDB8B37E9C6E6D635749DF3169D5F005D + Data: + Validators: CE25FBFF2E10C0D51AA1A07C064A96931BC8B297 + App: E0FBAFBF6FCED8B9786DDFEB1A0D4FA2501BADAD + }#F671D562C7B9242900A286E1882EE64E5556FE9E + Data{ + + }# + Commit{ + BlockID: F15AB8BEF9A6AAB07E457A6E16BC410546AA4DC6:1:D505DA273544 + Precommits: Vote{0:125B0E3C5512 90/00/2(Precommit) F15AB8BEF9A6 {/FE98E2B956F0.../}} + }#56FEF2EFDB8B37E9C6E6D635749DF3169D5F005D +}#F671D562C7B9242900A286E1882EE64E5556FE9E module=consensus +I[10-04|13:54:30.408] Executed block module=state height=91 validTxs=0 invalidTxs=0 +I[10-04|13:54:30.410] Committed state module=state height=91 txs=0 hash=E0FBAFBF6FCED8B9786DDFEB1A0D4FA2501BADAD +I[10-04|13:54:30.410] Recheck txs module=mempool numtxs=0 height=91 +``` + +## List of modules + +Here is the list of modules you may encounter in CometBFT's log and a +little overview what they do. + +- `abci-client` As mentioned in [Application Development Guide](../app-dev/abci-cli.md), CometBFT acts as an ABCI + client with respect to the application and maintains 3 connections: + mempool, consensus and query. The code used by CometBFT can + be found [here](https://github.com/cometbft/cometbft/blob/v0.38.x/abci/client). +- `blockchain` Provides storage, pool (a group of peers), and reactor + for both storing and exchanging blocks between peers. +- `consensus` The heart of CometBFT, which is the + implementation of the consensus algorithm. Includes two + "submodules": `wal` (write-ahead logging) for ensuring data + integrity and `replay` to replay blocks and messages on recovery + from a crash. +- `events` Simple event notification system. The list of events can be + found + [here](https://github.com/cometbft/cometbft/blob/v0.38.x/types/events.go). + You can subscribe to them by calling `subscribe` RPC method. Refer + to [RPC docs](./rpc.md) for additional information. +- `mempool` Mempool module handles all incoming transactions, whenever + they are coming from peers or the application. +- `p2p` Provides an abstraction around peer-to-peer communication. For + more details, please check out the + [README](https://github.com/cometbft/cometbft/blob/v0.38.x/p2p/README.md). +- `rpc` [CometBFT's RPC](./rpc.md). +- `rpc-server` RPC server. For implementation details, please read the + [doc.go](https://github.com/cometbft/cometbft/blob/v0.38.x/rpc/jsonrpc/doc.go). +- `state` Represents the latest state and execution submodule, which + executes blocks against the application. +- `types` A collection of the publicly exposed types and methods to + work with them. diff --git a/cometbft/v0.38/docs/core/light-client.mdx b/cometbft/v0.38/docs/core/light-client.mdx new file mode 100644 index 00000000..c64c9163 --- /dev/null +++ b/cometbft/v0.38/docs/core/light-client.mdx @@ -0,0 +1,69 @@ +--- +order: 13 +--- + +# Light Client + +Light clients are an important part of the complete blockchain system for most +applications. CometBFT provides unique speed and security properties for +light client applications. + +See our [light +package](https://pkg.go.dev/github.com/cometbft/cometbft/light?tab=doc). + +## Overview + +The objective of the light client protocol is to get a commit for a recent +block hash where the commit includes a majority of signatures from the last +known validator set. From there, all the application state is verifiable with +[merkle proofs](https://github.com/cometbft/cometbft/blob/v0.38.x/spec/core/encoding.md#iavl-tree). + +## Properties + +- You get the full collateralized security benefits of CometBFT; no + need to wait for confirmations. +- You get the full speed benefits of CometBFT; transactions + commit instantly. +- You can get the most recent version of the application state + non-interactively (without committing anything to the blockchain). For + example, this means that you can get the most recent value of a name from the + name-registry without worrying about fork censorship attacks, without posting + a commit and waiting for confirmations. It's fast, secure, and free! + +## Where to obtain trusted height & hash + +[Trust Options](https://pkg.go.dev/github.com/cometbft/cometbft/light?tab=doc#TrustOptions) + +One way to obtain semi-trusted hash & height is to query multiple full nodes +and compare their hashes: + +```bash +$ curl -s https://233.123.0.140:26657:26657/commit | jq "{height: .result.signed_header.header.height, hash: .result.signed_header.commit.block_id.hash}" +{ + "height": "273", + "hash": "188F4F36CBCD2C91B57509BBF231C777E79B52EE3E0D90D06B1A25EB16E6E23D" +} +``` + +## Running a light client as an HTTP proxy server + +CometBFT comes with a built-in `cometbft light` command, which can be used +to run a light client proxy server, verifying CometBFT RPC. All calls that +can be tracked back to a block header by a proof will be verified before +passing them back to the caller. Other than that, it will present the same +interface as a full CometBFT node. + +You can start the light client proxy server by running `cometbft light `, +with a variety of flags to specify the primary node, the witness nodes (which cross-check +the information provided by the primary), the hash and height of the trusted header, +and more. + +For example: + +```bash +$ cometbft light supernova -p tcp://233.123.0.140:26657 \ + -w tcp://179.63.29.15:26657,tcp://144.165.223.135:26657 \ + --height=10 --hash=37E9A6DD3FA25E83B22C18835401E8E56088D0D7ABC6FD99FCDC920DD76C1C57 +``` + +For additional options, run `cometbft light --help`. diff --git a/cometbft/v0.38/docs/core/mempool.mdx b/cometbft/v0.38/docs/core/mempool.mdx new file mode 100644 index 00000000..f86083ee --- /dev/null +++ b/cometbft/v0.38/docs/core/mempool.mdx @@ -0,0 +1,103 @@ +--- +order: 12 +--- + +# Mempool + +A mempool (a contraction of memory and pool) is a node’s data structure for +storing information on uncommitted transactions. It acts as a sort of waiting +room for transactions that have not yet been committed. + +CometBFT currently supports two types of mempools: `flood` and `nop`. + +## 1. Flood + +The `flood` mempool stores transactions in a concurrent linked list. When a new +transaction is received, it first checks if there's a space for it (`size` and +`max_txs_bytes` config options) and that it's not too big (`max_tx_bytes` config +option). Then, it checks if this transaction has already been seen before by using +an LRU cache (`cache_size` regulates the cache's size). If all checks pass and +the transaction is not in the cache (meaning it's new), the ABCI +[`CheckTxAsync`][1] method is called. The ABCI application validates the +transaction using its own rules. + +If the transaction is deemed valid by the ABCI application, it's added to the linked list. + +The mempool's name (`flood`) comes from the dissemination mechanism. When a new +transaction is added to the linked list, the mempool sends it to all connected +peers. Peers themselves gossip this transaction to their peers and so on. One +can say that each transaction "floods" the network, hence the name `flood`. + +Note there are experimental config options +`experimental_max_gossip_connections_to_persistent_peers` and +`experimental_max_gossip_connections_to_non_persistent_peers` to limit the +number of peers a transaction is broadcasted to. Also, you can turn off +broadcasting with `broadcast` config option. + +After each committed block, CometBFT rechecks all uncommitted transactions (can +be disabled with the `recheck` config option) by repeatedly calling the ABCI +`CheckTxAsync`. + +### Transaction ordering + +Currently, there's no ordering of transactions other than the order they've +arrived (via RPC or from other nodes). + +So the only way to specify the order is to send them to a single node. + +valA: + +- `tx1` +- `tx2` +- `tx3` + +If the transactions are split up across different nodes, there's no way to +ensure they are processed in the expected order. + +valA: + +- `tx1` +- `tx2` + +valB: + +- `tx3` + +If valB is the proposer, the order might be: + +- `tx3` +- `tx1` +- `tx2` + +If valA is the proposer, the order might be: + +- `tx1` +- `tx2` +- `tx3` + +That said, if the transactions contain some internal value, like an +order/nonce/sequence number, the application can reject transactions that are +out of order. So if a node receives `tx3`, then `tx1`, it can reject `tx3` and then +accept `tx1`. The sender can then retry sending `tx3`, which should probably be +rejected until the node has seen `tx2`. + +## 2. Nop + +`nop` (short for no operation) mempool is used when the ABCI application developer wants to +build their own mempool. When `type = "nop"`, transactions are not stored anywhere +and are not gossiped to other peers using the P2P network. + +Submitting a transaction via the existing RPC methods (`BroadcastTxSync`, +`BroadcastTxAsync`, and `BroadcastTxCommit`) will always result in an error. + +Because there's no way for the consensus to know if transactions are available +to be committed, the node will always create blocks, which can be empty +sometimes. Using `consensus.create_empty_blocks=false` is prohibited in such +cases. + +The ABCI application becomes responsible for storing, disseminating, and +proposing transactions using [`PrepareProposal`][2]. The concrete design is up +to the ABCI application developers. + +[1]: ../../spec/abci/abci++_methods.md#checktx +[2]: ../../spec/abci/abci++_methods.md#prepareproposal \ No newline at end of file diff --git a/cometbft/v0.38/docs/core/metrics.mdx b/cometbft/v0.38/docs/core/metrics.mdx new file mode 100644 index 00000000..5e97318f --- /dev/null +++ b/cometbft/v0.38/docs/core/metrics.mdx @@ -0,0 +1,75 @@ +--- +order: 5 +--- + +# Metrics + +CometBFT can report and serve the Prometheus metrics, which in their turn can +be consumed by Prometheus collector(s). + +This functionality is disabled by default. + +To enable the Prometheus metrics, set `instrumentation.prometheus=true` in your +config file. Metrics will be served under `/metrics` on 26660 port by default. +Listen address can be changed in the config file (see +`instrumentation.prometheus\_listen\_addr`). + +## List of available metrics + +The following metrics are available: + +| **Name** | **Type** | **Tags** | **Description** | +|--------------------------------------------|-----------|------------------|--------------------------------------------------------------------------------------------------------------------------------------------| +| abci\_connection\_method\_timing\_seconds | Histogram | method, type | Timings for each of the ABCI methods | +| blocksync\_syncing | Gauge | | Either 0 (not block syncing) or 1 (syncing) | +| consensus\_height | Gauge | | Height of the chain | +| consensus\_validators | Gauge | | Number of validators | +| consensus\_validators\_power | Gauge | | Total voting power of all validators | +| consensus\_validator\_power | Gauge | | Voting power of the node if in the validator set | +| consensus\_validator\_last\_signed\_height | Gauge | | Last height the node signed a block, if the node is a validator | +| consensus\_validator\_missed\_blocks | Gauge | | Total amount of blocks missed for the node, if the node is a validator | +| consensus\_missing\_validators | Gauge | | Number of validators who did not sign | +| consensus\_missing\_validators\_power | Gauge | | Total voting power of the missing validators | +| consensus\_byzantine\_validators | Gauge | | Number of validators who tried to double sign | +| consensus\_byzantine\_validators\_power | Gauge | | Total voting power of the byzantine validators | +| consensus\_block\_interval\_seconds | Histogram | | Time between this and last block (Block.Header.Time) in seconds | +| consensus\_rounds | Gauge | | Number of rounds | +| consensus\_num\_txs | Gauge | | Number of transactions | +| consensus\_total\_txs | Gauge | | Total number of transactions committed | +| consensus\_block\_parts | Counter | peer\_id | Number of blockparts transmitted by peer | +| consensus\_latest\_block\_height | Gauge | | /status sync\_info number | +| consensus\_block\_size\_bytes | Gauge | | Block size in bytes | +| consensus\_step\_duration | Histogram | step | Histogram of durations for each step in the consensus protocol | +| consensus\_round\_duration | Histogram | | Histogram of durations for all the rounds that have occurred since the process started | +| consensus\_block\_gossip\_parts\_received | Counter | matches\_current | Number of block parts received by the node | +| consensus\_quorum\_prevote\_delay | Gauge | | Interval in seconds between the proposal timestamp and the timestamp of the earliest prevote that achieved a quorum | +| consensus\_full\_prevote\_delay | Gauge | | Interval in seconds between the proposal timestamp and the timestamp of the latest prevote in a round where all validators voted | +| consensus\_vote\_extension\_receive\_count | Counter | status | Number of vote extensions received | +| consensus\_proposal\_receive\_count | Counter | status | Total number of proposals received by the node since process start | +| consensus\_proposal\_create\_count | Counter | | Total number of proposals created by the node since process start | +| consensus\_round\_voting\_power\_percent | Gauge | vote\_type | A value between 0 and 1.0 representing the percentage of the total voting power per vote type received within a round | +| consensus\_late\_votes | Counter | vote\_type | Number of votes received by the node since process start that correspond to earlier heights and rounds than this node is currently in. | +| p2p\_message\_send\_bytes\_total | Counter | message\_type | Number of bytes sent to all peers per message type | +| p2p\_message\_receive\_bytes\_total | Counter | message\_type | Number of bytes received from all peers per message type | +| p2p\_peers | Gauge | | Number of peers node's connected to | +| p2p\_peer\_receive\_bytes\_total | Counter | peer\_id, chID | Number of bytes per channel received from a given peer | +| p2p\_peer\_send\_bytes\_total | Counter | peer\_id, chID | Number of bytes per channel sent to a given peer | +| p2p\_peer\_pending\_send\_bytes | Gauge | peer\_id | Number of pending bytes to be sent to a given peer | +| p2p\_num\_txs | Gauge | peer\_id | Number of transactions submitted by each peer\_id | +| p2p\_pending\_send\_bytes | Gauge | peer\_id | Amount of data pending to be sent to peer | +| mempool\_size | Gauge | | Number of uncommitted transactions | +| mempool\_tx\_size\_bytes | Histogram | | Transaction sizes in bytes | +| mempool\_failed\_txs | Counter | | Number of failed transactions | +| mempool\_recheck\_times | Counter | | Number of transactions rechecked in the mempool | +| state\_block\_processing\_time | Histogram | | Time spent processing FinalizeBlock in ms | +| state\_consensus\_param\_updates | Counter | | Number of consensus parameter updates returned by the application since process start | +| state\_validator\_set\_updates | Counter | | Number of validator set updates returned by the application since process start | +| statesync\_syncing | Gauge | | Either 0 (not state syncing) or 1 (syncing) | + +## Useful queries + +Percentage of missing + byzantine validators: + +```md +((consensus\_byzantine\_validators\_power + consensus\_missing\_validators\_power) / consensus\_validators\_power) * 100 +``` diff --git a/cometbft/v0.38/docs/core/state-sync.mdx b/cometbft/v0.38/docs/core/state-sync.mdx new file mode 100644 index 00000000..7f5c5433 --- /dev/null +++ b/cometbft/v0.38/docs/core/state-sync.mdx @@ -0,0 +1,50 @@ +--- +order: 11 +--- + +# State Sync + +With block sync a node is downloading all of the data of an application from genesis and verifying it. +With state sync your node will download data related to the head or near the head of the chain and verify the data. +This leads to drastically shorter times for joining a network. + +## Using State Sync + +State sync will continuously work in the background to supply nodes with chunked data when bootstrapping. + +> NOTE: Before trying to use state sync, see if the application you are operating a node for supports it. + +Under the state sync section in `config.toml` you will find multiple settings that need to be configured in order for your node to use state sync. + +Lets breakdown the settings: + +- `enable`: Enable is to inform the node that you will be using state sync to bootstrap your node. +- `rpc_servers`: RPC servers are needed because state sync utilizes the light client for verification. + - 2 servers are required, more is always helpful. +- `temp_dir`: Temporary directory is store the chunks in the machines local storage, If nothing is set it will create a directory in `/tmp` + +The next information you will need to acquire it through publicly exposed RPC's or a block explorer which you trust. + +- `trust_height`: Trusted height defines at which height your node should trust the chain. +- `trust_hash`: Trusted hash is the hash in the `BlockID` corresponding to the trusted height. +- `trust_period`: Trust period is the period in which headers can be verified. + > :warning: This value should be significantly smaller than the unbonding period. + +If you are relying on publicly exposed RPC's to get the need information, you can use `curl` and [`jq`][jq]. + +Example: + +```bash +curl -s https://233.123.0.140:26657/commit | jq "{height: .result.signed_header.header.height, hash: .result.signed_header.commit.block_id.hash}" +``` + +The response will be: + +```json +{ + "height": "273", + "hash": "188F4F36CBCD2C91B57509BBF231C777E79B52EE3E0D90D06B1A25EB16E6E23D" +} +``` + +[jq]: https://jqlang.github.io/jq/ diff --git a/cometbft/v0.38/docs/explanation/core/metrics.md b/cometbft/v0.38/docs/explanation/core/metrics.md new file mode 100644 index 00000000..90bc8a86 --- /dev/null +++ b/cometbft/v0.38/docs/explanation/core/metrics.md @@ -0,0 +1,92 @@ +--- +order: 5 +--- + +# Metrics + +CometBFT can report and serve the Prometheus metrics, which in their turn can +be consumed by Prometheus collector(s). + +This functionality is disabled by default. + +To enable the Prometheus metrics, set `instrumentation.prometheus=true` in your +config file. Metrics will be served under `/metrics` on 26660 port by default. +Listen address can be changed in the config file (see +`instrumentation.prometheus\_listen\_addr`). + +## List of available metrics + +The following metrics are available: + +| **Name** | **Type** | **Tags** | **Description** | +| ------------------------------------------------------- | --------- | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------- | +| abci\_connection\_method\_timing\_seconds | Histogram | method, type | Timings for each of the ABCI methods | +| blocksync\_syncing | Gauge | | Either 0 (not block syncing) or 1 (syncing) | +| consensus\_height | Gauge | | Height of the chain | +| consensus\_validators | Gauge | | Number of validators | +| consensus\_validators\_power | Gauge | validator\_address | Total voting power of all validators | +| consensus\_validator\_power | Gauge | validator\_address | Voting power of the node if in the validator set | +| consensus\_validator\_last\_signed\_height | Gauge | validator\_address | Last height the node signed a block, if the node is a validator | +| consensus\_validator\_missed\_blocks | Gauge | | Total amount of blocks missed for the node, if the node is a validator | +| consensus\_missing\_validators | Gauge | | Number of validators who did not sign | +| consensus\_missing\_validators\_power | Gauge | | Total voting power of the missing validators | +| consensus\_byzantine\_validators | Gauge | | Number of validators who tried to double sign | +| consensus\_byzantine\_validators\_power | Gauge | | Total voting power of the byzantine validators | +| consensus\_block\_interval\_seconds | Histogram | | Time between this and last block (Block.Header.Time) in seconds | +| consensus\_rounds | Gauge | | Number of rounds | +| consensus\_num\_txs | Gauge | | Number of transactions | +| consensus\_total\_txs | Gauge | | Total number of transactions committed | +| consensus\_block\_parts | Counter | peer\_id | Number of blockparts transmitted by peer | +| consensus\_latest\_block\_height | Gauge | | /status sync\_info number | +| consensus\_block\_size\_bytes | Gauge | | Block size in bytes | +| consensus\_step\_duration\_seconds | Histogram | step | Histogram of durations for each step in the consensus protocol | +| consensus\_round\_duration\_seconds | Histogram | | Histogram of durations for all the rounds that have occurred since the process started | +| consensus\_block\_gossip\_parts\_received | Counter | matches\_current | Number of block parts received by the node | +| consensus\_quorum\_prevote\_delay | Gauge | proposer\_address | Interval in seconds between the proposal timestamp and the timestamp of the earliest prevote that achieved a quorum | +| consensus\_full\_prevote\_delay | Gauge | proposer\_address | Interval in seconds between the proposal timestamp and the timestamp of the latest prevote in a round where all validators voted | +| consensus\_vote\_extension\_receive\_count | Counter | status | Number of vote extensions received | +| consensus\_proposal\_receive\_count | Counter | status | Total number of proposals received by the node since process start | +| consensus\_proposal\_create\_count | Counter | | Total number of proposals created by the node since process start | +| consensus\_round\_voting\_power\_percent | Gauge | vote\_type | A value between 0 and 1.0 representing the percentage of the total voting power per vote type received within a round | +| consensus\_late\_votes | Counter | vote\_type | Number of votes received by the node since process start that correspond to earlier heights and rounds than this node is currently in. | +| consensus\_duplicate\_vote | Counter | | Number of times we received a duplicate vote. | +| consensus\_duplicate\_block\_part | Counter | | Number of times we received a duplicate block part. | +| consensus\_proposal\_timestamp\_difference | Histogram | is\_timely | Difference between the timestamp in the proposal message and the local time of the validator at the time it received the message. | +| p2p\_message\_send\_bytes\_total | Counter | message\_type | Number of bytes sent to all peers per message type | +| p2p\_message\_receive\_bytes\_total | Counter | message\_type | Number of bytes received from all peers per message type | +| p2p\_peers | Gauge | | Number of peers node's connected to | +| p2p\_peer\_pending\_send\_bytes | Gauge | peer\_id | Number of pending bytes to be sent to a given peer | +| p2p\_recv\_rate\_limiter\_delay | Counter | peer\_id | Time in seconds spent sleeping by the receive rate limiter, in seconds. | +| p2p\_send\_rate\_limiter\_delay | Counter | peer\_id | Time in seconds spent sleeping by the send rate limiter, in seconds. | +| mempool\_size | Gauge | | Number of uncommitted transactions in the mempool | +| mempool\_size\_bytes | Gauge | | Total size of the mempool in bytes | +| mempool\_tx\_size\_bytes | Histogram | | Histogram of transaction sizes in bytes | +| mempool\_evicted\_txs | Counter | | Number of transactions that make it into the mempool and were later evicted for being invalid | +| mempool\_failed\_txs | Counter | | Number of transactions that failed to make it into the mempool for being invalid | +| mempool\_rejected\_txs | Counter | | Number of transactions that failed to make it into the mempool due to resource limits | +| mempool\_recheck\_times | Counter | | Number of times transactions are rechecked in the mempool | +| mempool\_already\_received\_txs | Counter | | Number of times transactions were received more than once | +| mempool\_active\_outbound\_connections | Gauge | | Number of connections being actively used for gossiping transaction (experimental) | +| mempool\_recheck\_duration\_seconds | Gauge | | Cumulative time spent rechecking transactions | +| state\_consensus\_param\_updates | Counter | | Number of consensus parameter updates returned by the application since process start | +| state\_validator\_set\_updates | Counter | | Number of validator set updates returned by the application since process start | +| state\_pruning\_service\_block\_retain\_height | Gauge | | Accepted block retain height set by the data companion | +| state\_pruning\_service\_block\_results\_retain\_height | Gauge | | Accepted block results retain height set by the data companion | +| state\_pruning\_service\_tx\_indexer\_retain\_height | Gauge | | Accepted transactions indices retain height set by the data companion | +| state\_pruning\_service\_block\_indexer\_retain\_height | Gauge | | Accepted blocks indices retain height set by the data companion | +| state\_application\_block\_retain\_height | Gauge | | Accepted block retain height set by the application | +| state\_block\_store\_base\_height | Gauge | | First height at which a block is available | +| state\_abciresults\_base\_height | Gauge | | First height at which ABCI results are available | +| state\_tx\_indexer\_base\_height | Gauge | | First height at which tx indices are available | +| state\_block\_indexer\_base\_height | Gauge | | First height at which block indices are available | +| state\_store\_access\_duration\_seconds | Histogram | method | Duration of accesses to the state store labeled by which method was called on the store | +| state\_fire\_block\_events\_delay\_seconds | Gauge | | Duration of event firing related to a new block | +| statesync\_syncing | Gauge | | Either 0 (not state syncing) or 1 (syncing) | + +## Useful queries + +Percentage of missing + byzantine validators: + +```md +((consensus\_byzantine\_validators\_power + consensus\_missing\_validators\_power) / consensus\_validators\_power) * 100 +``` diff --git a/cometbft/v0.38/docs/guides/Creating-a-built-in-application-in-Go.mdx b/cometbft/v0.38/docs/guides/Creating-a-built-in-application-in-Go.mdx new file mode 100644 index 00000000..d07618fd --- /dev/null +++ b/cometbft/v0.38/docs/guides/Creating-a-built-in-application-in-Go.mdx @@ -0,0 +1,798 @@ +--- +order: 2 +--- + +# Creating a built-in application in Go + +## Guide Assumptions + +This guide is designed for beginners who want to get started with a CometBFT +application from scratch. It does not assume that you have any prior +experience with CometBFT. + +CometBFT is a service that provides a Byzantine Fault Tolerant consensus engine +for state-machine replication. The replicated state-machine, or "application", can be written +in any language that can send and receive protocol buffer messages in a client-server model. +Applications written in Go can also use CometBFT as a library and run the service in the same +process as the application. + +By following along this tutorial you will create a CometBFT application called kvstore, +a (very) simple distributed BFT key-value store. +The application will be written in Go and +some understanding of the Go programming language is expected. +If you have never written Go, you may want to go through [Learn X in Y minutes +Where X=Go](https://learnxinyminutes.com/docs/go/) first, to familiarize +yourself with the syntax. + +Note: Please use the latest released version of this guide and of CometBFT. +We strongly advise against using unreleased commits for your development. + +### Built-in app vs external app + +On the one hand, to get maximum performance you can run your application in +the same process as the CometBFT, as long as your application is written in Go. +[Cosmos SDK](https://github.com/cosmos/cosmos-sdk) is written +this way. +This is the approach followed in this tutorial. + +On the other hand, having a separate application might give you better security +guarantees as two processes would be communicating via established binary protocol. +CometBFT will not have access to application's state. +If that is the way you wish to proceed, use the [Creating an application in Go](./go.md) guide instead of this one. + +## 1.1 Installing Go + +Verify that you have the latest version of Go installed (refer to the [official guide for installing Go](https://golang.org/doc/install)): + +```bash +$ go version +go version go1.22.11 darwin/amd64 +``` + +## 1.2 Creating a new Go project + +We'll start by creating a new Go project. + +```bash +mkdir kvstore +``` + +Inside the example directory, create a `main.go` file with the following content: + +```go +package main + +import ( + "fmt" +) + +func main() { + fmt.Println("Hello, CometBFT") +} +``` + +When run, this should print "Hello, CometBFT" to the standard output. + +```bash +cd kvstore +$ go run main.go +Hello, CometBFT +``` + +We are going to use [Go modules](https://github.com/golang/go/wiki/Modules) for +dependency management, so let's start by including a dependency on the latest version of +CometBFT, `v0.38.0` in this example. + +```bash +go mod init kvstore +go get github.com/cometbft/cometbft@v0.38.0 +``` + +After running the above commands you will see two generated files, `go.mod` and `go.sum`. +The go.mod file should look similar to: + +```go +module kvstore + +go 1.22 + +require ( +github.com/cometbft/cometbft v0.38.0 +) +``` + +XXX: CometBFT `v0.38.0` uses a slightly outdated `gogoproto` library, which +may fail to compile with newer Go versions. To avoid any compilation errors, +upgrade `gogoproto` manually: + +```bash +go get github.com/cosmos/gogoproto@v1.4.11 +``` + +As you write the kvstore application, you can rebuild the binary by +pulling any new dependencies and recompiling it. + +```bash +go get +go build +``` + +## 1.3 Writing a CometBFT application + +CometBFT communicates with the application through the Application +BlockChain Interface (ABCI). The messages exchanged through the interface are +defined in the ABCI [protobuf +file](https://github.com/cometbft/cometbft/blob/v0.38.x/proto/tendermint/abci/types.proto). + +We begin by creating the basic scaffolding for an ABCI application by +creating a new type, `KVStoreApplication`, which implements the +methods defined by the `abcitypes.Application` interface. + +Create a file called `app.go` with the following contents: + +```go +package main + +import ( + abcitypes "github.com/cometbft/cometbft/abci/types" + "context" +) + +type KVStoreApplication struct{} + +var _ abcitypes.Application = (*KVStoreApplication)(nil) + +func NewKVStoreApplication() *KVStoreApplication { + return &KVStoreApplication{} +} + +func (app *KVStoreApplication) Info(_ context.Context, info *abcitypes.RequestInfo) (*abcitypes.ResponseInfo, error) { + return &abcitypes.ResponseInfo{}, nil +} + +func (app *KVStoreApplication) Query(_ context.Context, req *abcitypes.RequestQuery) (*abcitypes.ResponseQuery, error) { + return &abcitypes.ResponseQuery{}, nil +} + +func (app *KVStoreApplication) CheckTx(_ context.Context, check *abcitypes.RequestCheckTx) (*abcitypes.ResponseCheckTx, error) { + return &abcitypes.ResponseCheckTx{}, nil +} + +func (app *KVStoreApplication) InitChain(_ context.Context, chain *abcitypes.RequestInitChain) (*abcitypes.ResponseInitChain, error) { + return &abcitypes.ResponseInitChain{}, nil +} + +func (app *KVStoreApplication) PrepareProposal(_ context.Context, proposal *abcitypes.RequestPrepareProposal) (*abcitypes.ResponsePrepareProposal, error) { + return &abcitypes.ResponsePrepareProposal{}, nil +} + +func (app *KVStoreApplication) ProcessProposal(_ context.Context, proposal *abcitypes.RequestProcessProposal) (*abcitypes.ResponseProcessProposal, error) { + return &abcitypes.ResponseProcessProposal{}, nil +} + +func (app *KVStoreApplication) FinalizeBlock(_ context.Context, req *abcitypes.RequestFinalizeBlock) (*abcitypes.ResponseFinalizeBlock, error) { + return &abcitypes.ResponseFinalizeBlock{}, nil +} + +func (app KVStoreApplication) Commit(_ context.Context, commit *abcitypes.RequestCommit) (*abcitypes.ResponseCommit, error) { + return &abcitypes.ResponseCommit{}, nil +} + +func (app *KVStoreApplication) ListSnapshots(_ context.Context, snapshots *abcitypes.RequestListSnapshots) (*abcitypes.ResponseListSnapshots, error) { + return &abcitypes.ResponseListSnapshots{}, nil +} + +func (app *KVStoreApplication) OfferSnapshot(_ context.Context, snapshot *abcitypes.RequestOfferSnapshot) (*abcitypes.ResponseOfferSnapshot, error) { + return &abcitypes.ResponseOfferSnapshot{}, nil +} + +func (app *KVStoreApplication) LoadSnapshotChunk(_ context.Context, chunk *abcitypes.RequestLoadSnapshotChunk) (*abcitypes.ResponseLoadSnapshotChunk, error) { + return &abcitypes.ResponseLoadSnapshotChunk{}, nil +} + +func (app *KVStoreApplication) ApplySnapshotChunk(_ context.Context, chunk *abcitypes.RequestApplySnapshotChunk) (*abcitypes.ResponseApplySnapshotChunk, error) { + return &abcitypes.ResponseApplySnapshotChunk{Result: abcitypes.ResponseApplySnapshotChunk_ACCEPT}, nil +} + +func (app KVStoreApplication) ExtendVote(_ context.Context, extend *abcitypes.RequestExtendVote) (*abcitypes.ResponseExtendVote, error) { + return &abcitypes.ResponseExtendVote{}, nil +} + +func (app *KVStoreApplication) VerifyVoteExtension(_ context.Context, verify *abcitypes.RequestVerifyVoteExtension) (*abcitypes.ResponseVerifyVoteExtension, error) { + return &abcitypes.ResponseVerifyVoteExtension{}, nil +} +``` + +The types used here are defined in the CometBFT library and were added as a dependency +to the project when you ran `go get`. If your IDE is not recognizing the types, go ahead and run the command again. + +```bash +go get github.com/cometbft/cometbft@v0.38.0 +``` + +Now go back to the `main.go` and modify the `main` function so it matches the following, +where an instance of the `KVStoreApplication` type is created. + +```go +func main() { + fmt.Println("Hello, CometBFT") + + _ = NewKVStoreApplication() +} +``` + +You can recompile and run the application now by running `go get` and `go build`, but it does +not do anything. +So let's revisit the code adding the logic needed to implement our minimal key/value store +and to start it along with the CometBFT Service. + +### 1.3.1 Add a persistent data store + +Our application will need to write its state out to persistent storage so that it +can stop and start without losing all of its data. + +For this tutorial, we will use [BadgerDB](https://github.com/dgraph-io/badger), a +fast embedded key-value store. + +First, add Badger as a dependency of your go module using the `go get` command: + +`go get github.com/dgraph-io/badger/v3` + +Next, let's update the application and its constructor to receive a handle to the database, as follows: + +```go +type KVStoreApplication struct { + db *badger.DB + onGoingBlock *badger.Txn +} + +var _ abcitypes.Application = (*KVStoreApplication)(nil) + +func NewKVStoreApplication(db *badger.DB) *KVStoreApplication { + return &KVStoreApplication{db: db} +} +``` + +The `onGoingBlock` keeps track of the Badger transaction that will update the application's state when a block +is completed. Don't worry about it for now, we'll get to that later. + +Next, update the `import` stanza at the top to include the Badger library: + +```go +import( + "github.com/dgraph-io/badger/v3" + abcitypes "github.com/cometbft/cometbft/abci/types" +) +``` + +Finally, update the `main.go` file to invoke the updated constructor: + +```go + _ = NewKVStoreApplication(nil) +``` + +### 1.3.2 CheckTx + +When CometBFT receives a new transaction from a client, or from another full node, +CometBFT asks the application if the transaction is acceptable, using the `CheckTx` method. +Invalid transactions will not be shared with other nodes and will not become part of any blocks and, therefore, will not be executed by the application. + +In our application, a transaction is a string with the form `key=value`, indicating a key and value to write to the store. + +The most basic validation check we can perform is to check if the transaction conforms to the `key=value` pattern. +For that, let's add the following helper method to app.go: + +```go +func (app *KVStoreApplication) isValid(tx []byte) uint32 { + // check format + parts := bytes.Split(tx, []byte("=")) + if len(parts) != 2 { + return 1 + } + + return 0 +} +``` + +Now you can rewrite the `CheckTx` method to use the helper function: + +```go +func (app *KVStoreApplication) CheckTx(_ context.Context, check *abcitypes.RequestCheckTx) (*abcitypes.ResponseCheckTx, error) { + code := app.isValid(check.Tx) + return &abcitypes.ResponseCheckTx{Code: code}, nil +} +``` + +While this `CheckTx` is simple and only validates that the transaction is well-formed, +it is very common for `CheckTx` to make more complex use of the state of an application. +For example, you may refuse to overwrite an existing value, or you can associate +versions to the key/value pairs and allow the caller to specify a version to +perform a conditional update. + +Depending on the checks and on the conditions violated, the function may return +different values, but any response with a non-zero code will be considered invalid +by CometBFT. Our `CheckTx` logic returns 0 to CometBFT when a transaction passes +its validation checks. The specific value of the code is meaningless to CometBFT. +Non-zero codes are logged by CometBFT so applications can provide more specific +information on why the transaction was rejected. + +Note that `CheckTx` does not execute the transaction, it only verifies that the transaction could be executed. We do not know yet if the rest of the network has agreed to accept this transaction into a block. + +Finally, make sure to add the `bytes` package to the `import` stanza at the top of `app.go`: + +```go +import( + "bytes" + + "github.com/dgraph-io/badger/v3" + abcitypes "github.com/cometbft/cometbft/abci/types" +) +``` + +### 1.3.3 FinalizeBlock + +When the CometBFT consensus engine has decided on the block, the block is transferred to the +application via `FinalizeBlock`. +`FinalizeBlock` is an ABCI method introduced in CometBFT `v0.38.0`. This replaces the functionality provided previously (pre-`v0.38.0`) by the combination of ABCI methods `BeginBlock`, `DeliverTx`, and `EndBlock`. `FinalizeBlock`'s parameters are an aggregation of those in `BeginBlock`, `DeliverTx`, and `EndBlock`. + +This method is responsible for executing the block and returning a response to the consensus engine. +Providing a single `FinalizeBlock` method to signal the finalization of a block simplifies the ABCI interface and increases flexibility in the execution pipeline. + +The `FinalizeBlock` method executes the block, including any necessary transaction processing and state updates, and returns a `ResponseFinalizeBlock` object which contains any necessary information about the executed block. + +**Note:** `FinalizeBlock` only prepares the update to be made and does not change the state of the application. The state change is actually committed in a later stage i.e. in `commit` phase. + +Note that, to implement these calls in our application we're going to make use of Badger's transaction mechanism. We will always refer to these as Badger transactions, not to confuse them with the transactions included in the blocks delivered by CometBFT, the _application transactions_. + +First, let's create a new Badger transaction during `FinalizeBlock`. All application transactions in the current block will be executed within this Badger transaction. +Next, let's modify `FinalizeBlock` to add the `key` and `value` to the Badger transaction every time our application processes a new application transaction from the list received through `RequestFinalizeBlock`. + +Note that we check the validity of the transaction _again_ during `FinalizeBlock`. + +```go +func (app *KVStoreApplication) FinalizeBlock(_ context.Context, req *abcitypes.RequestFinalizeBlock) (*abcitypes.ResponseFinalizeBlock, error) { + var txs = make([]*abcitypes.ExecTxResult, len(req.Txs)) + + app.onGoingBlock = app.db.NewTransaction(true) + for i, tx := range req.Txs { + if code := app.isValid(tx); code != 0 { + log.Printf("Error: invalid transaction index %v", i) + txs[i] = &abcitypes.ExecTxResult{Code: code} + } else { + parts := bytes.SplitN(tx, []byte("="), 2) + key, value := parts[0], parts[1] + log.Printf("Adding key %s with value %s", key, value) + + if err := app.onGoingBlock.Set(key, value); err != nil { + log.Panicf("Error writing to database, unable to execute tx: %v", err) + } + + log.Printf("Successfully added key %s with value %s", key, value) + + txs[i] = &abcitypes.ExecTxResult{} + } + } + + return &abcitypes.ResponseFinalizeBlock{ + TxResults: txs, + }, nil +} +``` + +Transactions are not guaranteed to be valid when they are delivered to an application, even if they were valid when they were proposed. + +This can happen if the application state is used to determine transaction validity. +The application state may have changed between the initial execution of `CheckTx` and the transaction delivery in `FinalizeBlock` in a way that rendered the transaction no longer valid. + +**Note** that `FinalizeBlock` cannot yet commit the Badger transaction we were building during the block execution. + +Other methods, such as `Query`, rely on a consistent view of the application's state, the application should only update its state by committing the Badger transactions when the full block has been delivered and the Commit method is invoked. + +The `Commit` method tells the application to make permanent the effects of +the application transactions. +Let's update the method to terminate the pending Badger transaction and +persist the resulting state: + +```go +func (app KVStoreApplication) Commit(_ context.Context, commit *abcitypes.RequestCommit) (*abcitypes.ResponseCommit, error) { + return &abcitypes.ResponseCommit{}, app.onGoingBlock.Commit() +} +``` + +Finally, make sure to add the log library to the `import` stanza as well: + +```go +import ( + "bytes" + "log" + + "github.com/dgraph-io/badger/v3" + abcitypes "github.com/cometbft/cometbft/abci/types" +) +``` + +You may have noticed that the application we are writing will crash if it receives +an unexpected error from the Badger database during the `FinalizeBlock` or `Commit` methods. +This is not an accident. If the application received an error from the database, there +is no deterministic way for it to make progress so the only safe option is to terminate. +Once the application is restarted, the transactions in the block that failed execution will +be re-executed and should succeed if the Badger error was transient. + +### 1.3.4 Query + +When a client tries to read some information from the `kvstore`, the request will be +handled in the `Query` method. To do this, let's rewrite the `Query` method in `app.go`: + +```go +func (app *KVStoreApplication) Query(_ context.Context, req *abcitypes.RequestQuery) (*abcitypes.ResponseQuery, error) { + resp := abcitypes.ResponseQuery{Key: req.Data} + + dbErr := app.db.View(func(txn *badger.Txn) error { + item, err := txn.Get(req.Data) + if err != nil { + if err != badger.ErrKeyNotFound { + return err + } + resp.Log = "key does not exist" + return nil + } + + return item.Value(func(val []byte) error { + resp.Log = "exists" + resp.Value = val + return nil + }) + }) + if dbErr != nil { + log.Panicf("Error reading database, unable to execute query: %v", dbErr) + } + return &resp, nil +} +``` + +Since it reads only committed data from the store, transactions that are part of a block +that is being processed are not reflected in the query result. + +### 1.3.5 PrepareProposal and ProcessProposal + +`PrepareProposal` and `ProcessProposal` are methods introduced in CometBFT v0.37.0 +to give the application more control over the construction and processing of transaction blocks. + +When CometBFT sees that valid transactions (validated through `CheckTx`) are available to be +included in blocks, it groups some of these transactions and then gives the application a chance +to modify the group by invoking `PrepareProposal`. + +The application is free to modify the group before returning from the call, as long as the resulting set +does not use more bytes than `RequestPrepareProposal.max_tx_bytes` +For example, the application may reorder, add, or even remove transactions from the group to improve the +execution of the block once accepted. + +In the following code, the application simply returns the unmodified group of transactions: + +```go +func (app *KVStoreApplication) PrepareProposal(_ context.Context, proposal *abcitypes.RequestPrepareProposal) (*abcitypes.ResponsePrepareProposal, error) { + return &abcitypes.ResponsePrepareProposal{Txs: proposal.Txs}, nil +} +``` + +Once a proposed block is received by a node, the proposal is passed to the application to give +its blessing before voting to accept the proposal. + +This mechanism may be used for different reasons, for example to deal with blocks manipulated +by malicious nodes, in which case the block should not be considered valid. + +The following code simply accepts all proposals: + +```go +func (app *KVStoreApplication) ProcessProposal(_ context.Context, proposal *abcitypes.RequestProcessProposal) (*abcitypes.ResponseProcessProposal, error) { + return &abcitypes.ResponseProcessProposal{Status: abcitypes.ResponseProcessProposal_ACCEPT}, nil +} +``` + +## 1.4 Starting an application and a CometBFT instance in the same process + +Now that we have the basic functionality of our application in place, let's put +it all together inside of our `main.go` file. + +Change the contents of your `main.go` file to the following. + +```go +package main + +import ( + "flag" + "fmt" + "github.com/cometbft/cometbft/p2p" + "github.com/cometbft/cometbft/privval" + "github.com/cometbft/cometbft/proxy" + "log" + "os" + "os/signal" + "path/filepath" + "syscall" + + "github.com/dgraph-io/badger/v3" + "github.com/spf13/viper" + cfg "github.com/cometbft/cometbft/config" + cmtflags "github.com/cometbft/cometbft/libs/cli/flags" + cmtlog "github.com/cometbft/cometbft/libs/log" + nm "github.com/cometbft/cometbft/node" +) + +var homeDir string + +func init() { + flag.StringVar(&homeDir, "cmt-home", "", "Path to the CometBFT config directory (if empty, uses $HOME/.cometbft)") +} + +func main() { + flag.Parse() + if homeDir == "" { + homeDir = os.ExpandEnv("$HOME/.cometbft") + } + + config := cfg.DefaultConfig() + config.SetRoot(homeDir) + viper.SetConfigFile(fmt.Sprintf("%s/%s", homeDir, "config/config.toml")) + + if err := viper.ReadInConfig(); err != nil { + log.Fatalf("Reading config: %v", err) + } + if err := viper.Unmarshal(config); err != nil { + log.Fatalf("Decoding config: %v", err) + } + if err := config.ValidateBasic(); err != nil { + log.Fatalf("Invalid configuration data: %v", err) + } + dbPath := filepath.Join(homeDir, "badger") + db, err := badger.Open(badger.DefaultOptions(dbPath)) + + if err != nil { + log.Fatalf("Opening database: %v", err) + } + defer func() { + if err := db.Close(); err != nil { + log.Printf("Closing database: %v", err) + } + }() + + app := NewKVStoreApplication(db) + + pv := privval.LoadFilePV( + config.PrivValidatorKeyFile(), + config.PrivValidatorStateFile(), + ) + + nodeKey, err := p2p.LoadNodeKey(config.NodeKeyFile()) + if err != nil { + log.Fatalf("failed to load node's key: %v", err) + } + + logger := cmtlog.NewTMLogger(cmtlog.NewSyncWriter(os.Stdout)) + logger, err = cmtflags.ParseLogLevel(config.LogLevel, logger, cfg.DefaultLogLevel) + + if err != nil { + log.Fatalf("failed to parse log level: %v", err) + } + + node, err := nm.NewNode( + config, + pv, + nodeKey, + proxy.NewLocalClientCreator(app), + nm.DefaultGenesisDocProviderFunc(config), + cfg.DefaultDBProvider, + nm.DefaultMetricsProvider(config.Instrumentation), + logger, + ) + + if err != nil { + log.Fatalf("Creating node: %v", err) + } + + node.Start() + defer func() { + node.Stop() + node.Wait() + }() + + c := make(chan os.Signal, 1) + signal.Notify(c, os.Interrupt, syscall.SIGTERM) + <-c +} +``` + +This is a huge blob of code, so let's break it down into pieces. + +First, we use [viper](https://github.com/spf13/viper) to load the CometBFT configuration files, which we will generate later: + +```go +config := cfg.DefaultValidatorConfig() + +config.SetRoot(homeDir) + +viper.SetConfigFile(fmt.Sprintf("%s/%s", homeDir, "config/config.toml")) +if err := viper.ReadInConfig(); err != nil { + log.Fatalf("Reading config: %v", err) +} +if err := viper.Unmarshal(config); err != nil { + log.Fatalf("Decoding config: %v", err) +} +if err := config.ValidateBasic(); err != nil { + log.Fatalf("Invalid configuration data: %v", err) +} +``` + +Next, we initialize the Badger database and create an app instance. + +```go +dbPath := filepath.Join(homeDir, "badger") +db, err := badger.Open(badger.DefaultOptions(dbPath)) +if err != nil { + log.Fatalf("Opening database: %v", err) +} +defer func() { + if err := db.Close(); err != nil { + log.Fatalf("Closing database: %v", err) + } +}() + +app := NewKVStoreApplication(db) +``` + +We use `FilePV`, which is a private validator (i.e. thing which signs consensus +messages). Normally, you would use `SignerRemote` to connect to an external +[HSM](https://kb.certus.one/hsm.html). + +```go +pv := privval.LoadFilePV( + config.PrivValidatorKeyFile(), + config.PrivValidatorStateFile(), +) +``` + +`nodeKey` is needed to identify the node in a p2p network. + +```go +nodeKey, err := p2p.LoadNodeKey(config.NodeKeyFile()) +if err != nil { + return nil, fmt.Errorf("failed to load node's key: %w", err) +} +``` + +Now we have everything set up to run the CometBFT node. We construct +a node by passing it the configuration, the logger, a handle to our application and +the genesis information: + +```go +node, err := nm.NewNode( + config, + pv, + nodeKey, + proxy.NewLocalClientCreator(app), + nm.DefaultGenesisDocProviderFunc(config), + cfg.DefaultDBProvider, + nm.DefaultMetricsProvider(config.Instrumentation), + logger) + +if err != nil { + log.Fatalf("Creating node: %v", err) +} +``` + +Finally, we start the node, i.e., the CometBFT service inside our application: + +```go +node.Start() +defer func() { + node.Stop() + node.Wait() +}() +``` + +The additional logic at the end of the file allows the program to catch SIGTERM. This means that the node can shut down gracefully when an operator tries to kill the program: + +```go +c := make(chan os.Signal, 1) +signal.Notify(c, os.Interrupt, syscall.SIGTERM) +<-c +``` + +## 1.5 Initializing and Running + +Our application is almost ready to run, but first we'll need to populate the CometBFT configuration files. +The following command will create a `cometbft-home` directory in your project and add a basic set of configuration files in `cometbft-home/config/`. +For more information on what these files contain see [the configuration documentation](https://github.com/cometbft/cometbft/blob/v0.38.x/docs/core/configuration.md). + +From the root of your project, run: + +```bash +go run github.com/cometbft/cometbft/cmd/cometbft@v0.38.0 init --home /tmp/cometbft-home +``` + +You should see an output similar to the following: + +```bash +I[2023-25-04|09:06:34.444] Generated private validator module=main keyFile=/tmp/cometbft-home/config/priv_validator_key.json stateFile=/tmp/cometbft-home/data/priv_validator_state.json +I[2023-25-04|09:06:34.444] Generated node key module=main path=/tmp/cometbft-home/config/node_key.json +I[2023-25-04|09:06:34.444] Generated genesis file module=main path=/tmp/cometbft-home/config/genesis.json +``` + +Now rebuild the app: + +```bash +go build -mod=mod # use -mod=mod to automatically refresh the dependencies +``` + +Everything is now in place to run your application. Run: + +```bash +./kvstore -cmt-home /tmp/cometbft-home +``` + +The application will start and you should see a continuous output starting with: + +```bash +badger 2023-04-25 09:08:50 INFO: All 0 tables opened in 0s +badger 2023-04-25 09:08:50 INFO: Discard stats nextEmptySlot: 0 +badger 2023-04-25 09:08:50 INFO: Set nextTxnTs to 0 +I[2023-04-25|09:08:50.085] service start module=proxy msg="Starting multiAppConn service" impl=multiAppConn +I[2023-04-25|09:08:50.085] service start module=abci-client connection=query msg="Starting localClient service" impl=localClient +I[2023-04-25|09:08:50.085] service start module=abci-client connection=snapshot msg="Starting localClient service" impl=localClient +... +``` + +More importantly, the application using CometBFT is producing blocks 🎉🎉 and you can see this reflected in the log output in lines like this: + +```bash +I[2023-04-25|09:08:52.147] received proposal module=consensus proposal="Proposal{2/0 (F518444C0E348270436A73FD0F0B9DFEA758286BEB29482F1E3BEA75330E825C:1:C73D3D1273F2, -1) AD19AE292A45 @ 2023-04-25T12:08:52.143393Z}" +I[2023-04-25|09:08:52.152] received complete proposal block module=consensus height=2 hash=F518444C0E348270436A73FD0F0B9DFEA758286BEB29482F1E3BEA75330E825C +I[2023-04-25|09:08:52.160] finalizing commit of block module=consensus height=2 hash=F518444C0E348270436A73FD0F0B9DFEA758286BEB29482F1E3BEA75330E825C root= num_txs=0 +I[2023-04-25|09:08:52.167] executed block module=state height=2 num_valid_txs=0 num_invalid_txs=0 +I[2023-04-25|09:08:52.171] committed state module=state height=2 num_txs=0 app_hash= +``` + +The blocks, as you can see from the `num_valid_txs=0` part, are empty, but let's remedy that next. + +## 1.6 Using the application + +Let's try submitting a transaction to our new application. +Open another terminal window and run the following curl command: + +```bash +curl -s 'localhost:26657/broadcast_tx_commit?tx="cometbft=rocks"' +``` + +If everything went well, you should see a response indicating which height the +transaction was included in the blockchain. + +Finally, let's make sure that transaction really was persisted by the application. +Run the following command: + +```bash +curl -s 'localhost:26657/abci_query?data="cometbft"' +``` + +Let's examine the response object that this request returns. +The request returns a `json` object with a `key` and `value` field set. + +```json +... + "key": "dGVuZGVybWludA==", + "value": "cm9ja3M=", +... +``` + +Those values don't look like the `key` and `value` we sent to CometBFT. +What's going on here? + +The response contains a `base64` encoded representation of the data we submitted. +To get the original value out of this data, we can use the `base64` command line utility: + +```bash +echo "cm9ja3M=" | base64 -d +``` + +## Outro + +Hope you could run everything smoothly. If you have any difficulties running through this tutorial, reach out to us via [discord](https://discord.com/invite/interchain) or open a new [issue](https://github.com/cometbft/cometbft/issues/new/choose) on Github. diff --git a/cometbft/v0.38/docs/guides/Creating-an-application-in-Go.mdx b/cometbft/v0.38/docs/guides/Creating-an-application-in-Go.mdx new file mode 100644 index 00000000..5da93607 --- /dev/null +++ b/cometbft/v0.38/docs/guides/Creating-an-application-in-Go.mdx @@ -0,0 +1,710 @@ +--- +order: 1 +--- + +# Creating an application in Go + +## Guide Assumptions + +This guide is designed for beginners who want to get started with a CometBFT +application from scratch. It does not assume that you have any prior +experience with CometBFT. + +CometBFT is a service that provides a Byzantine Fault Tolerant consensus engine +for state-machine replication. The replicated state-machine, or "application", can be written +in any language that can send and receive protocol buffer messages in a client-server model. +Applications written in Go can also use CometBFT as a library and run the service in the same +process as the application. + +By following along this tutorial you will create a CometBFT application called kvstore, +a (very) simple distributed BFT key-value store. +The application will be written in Go and +some understanding of the Go programming language is expected. +If you have never written Go, you may want to go through [Learn X in Y minutes +Where X=Go](https://learnxinyminutes.com/docs/go/) first, to familiarize +yourself with the syntax. + +Note: Please use the latest released version of this guide and of CometBFT. +We strongly advise against using unreleased commits for your development. + +### Built-in app vs external app + +On the one hand, to get maximum performance you can run your application in +the same process as the CometBFT, as long as your application is written in Go. +[Cosmos SDK](https://github.com/cosmos/cosmos-sdk) is written +this way. +If that is the way you wish to proceed, use the [Creating a built-in application in Go](./go-built-in.md) guide instead of this one. + +On the other hand, having a separate application might give you better security +guarantees as two processes would be communicating via established binary protocol. +CometBFT will not have access to application's state. +This is the approach followed in this tutorial. + +## 1.1 Installing Go + +Verify that you have the latest version of Go installed (refer to the [official guide for installing Go](https://golang.org/doc/install)): + +```bash +$ go version +go version go1.22.11 darwin/amd64 +``` + +## 1.2 Creating a new Go project + +We'll start by creating a new Go project. + +```bash +mkdir kvstore +``` + +Inside the example directory, create a `main.go` file with the following content: + +```go +package main + +import ( + "fmt" +) + +func main() { + fmt.Println("Hello, CometBFT") +} +``` + +When run, this should print "Hello, CometBFT" to the standard output. + +```bash +cd kvstore +$ go run main.go +Hello, CometBFT +``` + +We are going to use [Go modules](https://github.com/golang/go/wiki/Modules) for +dependency management, so let's start by including a dependency on the latest version of +CometBFT, `v0.38.0` in this example. + +```bash +go mod init kvstore +go get github.com/cometbft/cometbft@v0.38.0 +``` + +After running the above commands you will see two generated files, `go.mod` and `go.sum`. +The go.mod file should look similar to: + +```go +module kvstore + +go 1.22 + +require ( +github.com/cometbft/cometbft v0.38.0 +) +``` + +XXX: CometBFT `v0.38.0` uses a slightly outdated `gogoproto` library, which +may fail to compile with newer Go versions. To avoid any compilation errors, +upgrade `gogoproto` manually: + +```bash +go get github.com/cosmos/gogoproto@v1.4.11 +``` + +As you write the kvstore application, you can rebuild the binary by +pulling any new dependencies and recompiling it. + +```bash +go get +go build +``` + +## 1.3 Writing a CometBFT application + +CometBFT communicates with the application through the Application +BlockChain Interface (ABCI). The messages exchanged through the interface are +defined in the ABCI [protobuf +file](https://github.com/cometbft/cometbft/blob/v0.38.x/proto/tendermint/abci/types.proto). + +We begin by creating the basic scaffolding for an ABCI application by +creating a new type, `KVStoreApplication`, which implements the +methods defined by the `abcitypes.Application` interface. + +Create a file called `app.go` with the following contents: + +```go +package main + +import ( + abcitypes "github.com/cometbft/cometbft/abci/types" + "context" +) + +type KVStoreApplication struct{} + +var _ abcitypes.Application = (*KVStoreApplication)(nil) + +func NewKVStoreApplication() *KVStoreApplication { + return &KVStoreApplication{} +} + +func (app *KVStoreApplication) Info(_ context.Context, info *abcitypes.RequestInfo) (*abcitypes.ResponseInfo, error) { + return &abcitypes.ResponseInfo{}, nil +} + +func (app *KVStoreApplication) Query(_ context.Context, req *abcitypes.RequestQuery) (*abcitypes.ResponseQuery, error) { + return &abcitypes.ResponseQuery{}, nil +} + +func (app *KVStoreApplication) CheckTx(_ context.Context, check *abcitypes.RequestCheckTx) (*abcitypes.ResponseCheckTx, error) { + return &abcitypes.ResponseCheckTx{Code: code}, nil +} + +func (app *KVStoreApplication) InitChain(_ context.Context, chain *abcitypes.RequestInitChain) (*abcitypes.ResponseInitChain, error) { + return &abcitypes.ResponseInitChain{}, nil +} + +func (app *KVStoreApplication) PrepareProposal(_ context.Context, proposal *abcitypes.RequestPrepareProposal) (*abcitypes.ResponsePrepareProposal, error) { + return &abcitypes.ResponsePrepareProposal{}, nil +} + +func (app *KVStoreApplication) ProcessProposal(_ context.Context, proposal *abcitypes.RequestProcessProposal) (*abcitypes.ResponseProcessProposal, error) { + return &abcitypes.ResponseProcessProposal{}, nil +} + +func (app *KVStoreApplication) FinalizeBlock(_ context.Context, req *abcitypes.RequestFinalizeBlock) (*abcitypes.ResponseFinalizeBlock, error) { + return &abcitypes.ResponseFinalizeBlock{}, nil +} + +func (app KVStoreApplication) Commit(_ context.Context, commit *abcitypes.RequestCommit) (*abcitypes.ResponseCommit, error) { + return &abcitypes.ResponseCommit{}, nil +} + +func (app *KVStoreApplication) ListSnapshots(_ context.Context, snapshots *abcitypes.RequestListSnapshots) (*abcitypes.ResponseListSnapshots, error) { + return &abcitypes.ResponseListSnapshots{}, nil +} + +func (app *KVStoreApplication) OfferSnapshot(_ context.Context, snapshot *abcitypes.RequestOfferSnapshot) (*abcitypes.ResponseOfferSnapshot, error) { + return &abcitypes.ResponseOfferSnapshot{}, nil +} + +func (app *KVStoreApplication) LoadSnapshotChunk(_ context.Context, chunk *abcitypes.RequestLoadSnapshotChunk) (*abcitypes.ResponseLoadSnapshotChunk, error) { + return &abcitypes.ResponseLoadSnapshotChunk{}, nil +} + +func (app *KVStoreApplication) ApplySnapshotChunk(_ context.Context, chunk *abcitypes.RequestApplySnapshotChunk) (*abcitypes.ResponseApplySnapshotChunk, error) { + + return &abcitypes.ResponseApplySnapshotChunk{Result: abcitypes.ResponseApplySnapshotChunk_ACCEPT}, nil +} + +func (app KVStoreApplication) ExtendVote(_ context.Context, extend *abcitypes.RequestExtendVote) (*abcitypes.ResponseExtendVote, error) { + return &abcitypes.ResponseExtendVote{}, nil +} + +func (app *KVStoreApplication) VerifyVoteExtension(_ context.Context, verify *abcitypes.RequestVerifyVoteExtension) (*abcitypes.ResponseVerifyVoteExtension, error) { + return &abcitypes.ResponseVerifyVoteExtension{}, nil +} +``` + +The types used here are defined in the CometBFT library and were added as a dependency +to the project when you ran `go get`. If your IDE is not recognizing the types, go ahead and run the command again. + +```bash +go get github.com/cometbft/cometbft@v0.38.0 +``` + +Now go back to the `main.go` and modify the `main` function so it matches the following, +where an instance of the `KVStoreApplication` type is created. + +```go +func main() { + fmt.Println("Hello, CometBFT") + + _ = NewKVStoreApplication() +} +``` + +You can recompile and run the application now by running `go get` and `go build`, but it does +not do anything. +So let's revisit the code adding the logic needed to implement our minimal key/value store +and to start it along with the CometBFT Service. + +### 1.3.1 Add a persistent data store + +Our application will need to write its state out to persistent storage so that it +can stop and start without losing all of its data. + +For this tutorial, we will use [BadgerDB](https://github.com/dgraph-io/badger), a +a fast embedded key-value store. + +First, add Badger as a dependency of your go module using the `go get` command: + +`go get github.com/dgraph-io/badger/v3` + +Next, let's update the application and its constructor to receive a handle to the database, as follows: + +```go +type KVStoreApplication struct { + db *badger.DB + onGoingBlock *badger.Txn +} + +var _ abcitypes.Application = (*KVStoreApplication)(nil) + +func NewKVStoreApplication(db *badger.DB) *KVStoreApplication { + return &KVStoreApplication{db: db} +} +``` + +The `onGoingBlock` keeps track of the Badger transaction that will update the application's state when a block +is completed. Don't worry about it for now, we'll get to that later. + +Next, update the `import` stanza at the top to include the Badger library: + +```go +import( + "github.com/dgraph-io/badger/v3" + abcitypes "github.com/cometbft/cometbft/abci/types" +) +``` + +Finally, update the `main.go` file to invoke the updated constructor: + +```go +_ = NewKVStoreApplication(nil) +``` + +### 1.3.2 CheckTx + +When CometBFT receives a new transaction from a client, or from another full node, +CometBFT asks the application if the transaction is acceptable, using the `CheckTx` method. +Invalid transactions will not be shared with other nodes and will not become part of any blocks and, therefore, will not be executed by the application. + +In our application, a transaction is a string with the form `key=value`, indicating a key and value to write to the store. + +The most basic validation check we can perform is to check if the transaction conforms to the `key=value` pattern. +For that, let's add the following helper method to app.go: + +```go +func (app *KVStoreApplication) isValid(tx []byte) uint32 { + // check format + parts := bytes.Split(tx, []byte("=")) + if len(parts) != 2 { + return 1 + } + return 0 +} +``` + +Now you can rewrite the `CheckTx` method to use the helper function: + +```go +func (app *KVStoreApplication) CheckTx(_ context.Context, check *abcitypes.RequestCheckTx) (*abcitypes.ResponseCheckTx, error) { + code := app.isValid(check.Tx) + return &abcitypes.ResponseCheckTx{Code: code}, nil +} +``` + +While this `CheckTx` is simple and only validates that the transaction is well-formed, +it is very common for `CheckTx` to make more complex use of the state of an application. +For example, you may refuse to overwrite an existing value, or you can associate +versions to the key/value pairs and allow the caller to specify a version to +perform a conditional update. + +Depending on the checks and on the conditions violated, the function may return +different values, but any response with a non-zero code will be considered invalid +by CometBFT. Our `CheckTx` logic returns 0 to CometBFT when a transaction passes +its validation checks. The specific value of the code is meaningless to CometBFT. +Non-zero codes are logged by CometBFT so applications can provide more specific +information on why the transaction was rejected. + +Note that `CheckTx` does not execute the transaction, it only verifies that that the transaction could be executed. We do not know yet if the rest of the network has agreed to accept this transaction into a block. + +Finally, make sure to add the bytes package to the `import` stanza at the top of `app.go`: + +```go +import( + "bytes" + + "github.com/dgraph-io/badger/v3" + abcitypes "github.com/cometbft/cometbft/abci/types" +) +``` + +### 1.3.3 FinalizeBlock + +When the CometBFT consensus engine has decided on the block, the block is transferred to the +application via the `FinalizeBlock` method. +`FinalizeBlock` is an ABCI method introduced in CometBFT `v0.38.0`. This replaces the functionality provided previously (pre-`v0.38.0`) by the combination of ABCI methods `BeginBlock`, `DeliverTx`, and `EndBlock`. +`FinalizeBlock`'s parameters are an aggregation of those in `BeginBlock`, `DeliverTx`, and `EndBlock`. + +This method is responsible for executing the block and returning a response to the consensus engine. +Providing a single `FinalizeBlock` method to signal the finalization of a block simplifies the ABCI interface and increases flexibility in the execution pipeline. + +The `FinalizeBlock` method executes the block, including any necessary transaction processing and state updates, and returns a `ResponseFinalizeBlock` object which contains any necessary information about the executed block. + +**Note:** `FinalizeBlock` only prepares the update to be made and does not change the state of the application. The state change is actually committed in a later stage i.e. in `commit` phase. + +Note that, to implement these calls in our application we're going to make use of Badger's transaction mechanism. We will always refer to these as Badger transactions, not to confuse them with the transactions included in the blocks delivered by CometBFT, the _application transactions_. + +First, let's create a new Badger transaction during `FinalizeBlock`. All application transactions in the current block will be executed within this Badger transaction. +Next, let's modify `FinalizeBlock` to add the `key` and `value` to the database transaction every time our application processes a new application transaction from the list received through `RequestFinalizeBlock`. + +Note that we check the validity of the transaction _again_ during `FinalizeBlock`. + +```go +func (app *KVStoreApplication) FinalizeBlock(_ context.Context, req *abcitypes.RequestFinalizeBlock) (*abcitypes.ResponseFinalizeBlock, error) { + var txs = make([]*abcitypes.ExecTxResult, len(req.Txs)) + + app.onGoingBlock = app.db.NewTransaction(true) + for i, tx := range req.Txs { + if code := app.isValid(tx); code != 0 { + log.Printf("Error in tx in if") + txs[i] = &abcitypes.ExecTxResult{Code: code} + } else { + parts := bytes.SplitN(tx, []byte("="), 2) + key, value := parts[0], parts[1] + log.Printf("Adding key %s with value %s", key, value) + + if err := app.onGoingBlock.Set(key, value); err != nil { + log.Panicf("Error writing to database, unable to execute tx: %v", err) + } + log.Printf("Successfully added key %s with value %s", key, value) + + txs[i] = &abcitypes.ExecTxResult{} + } + } + + return &abcitypes.ResponseFinalizeBlock{ + TxResults: txs, + }, nil +} +``` + +Transactions are not guaranteed to be valid when they are delivered to an application, even if they were valid when they were proposed. + +This can happen if the application state is used to determine transaction validity. The application state may have changed between the initial execution of `CheckTx` and the transaction delivery in `FinalizeBlock` in a way that rendered the transaction no longer valid. + +**Note** that `FinalizeBlock` cannot yet commit the Badger transaction we were building during the block execution. + +Other methods, such as `Query`, rely on a consistent view of the application's state, the application should only update its state by committing the Badger transactions when the full block has been delivered and the `Commit` method is invoked. + +The `Commit` method tells the application to make permanent the effects of the application transactions. +Let's update the method to terminate the pending Badger transaction and persist the resulting state: + +```go +func (app KVStoreApplication) Commit(_ context.Context, commit *abcitypes.RequestCommit) (*abcitypes.ResponseCommit, error) { + return &abcitypes.ResponseCommit{}, app.onGoingBlock.Commit() +} +``` + +Finally, make sure to add the log library to the `import` stanza as well: + +```go +import ( + "bytes" + "log" + + "github.com/dgraph-io/badger/v3" + abcitypes "github.com/cometbft/cometbft/abci/types" +) +``` + +You may have noticed that the application we are writing will crash if it receives +an unexpected error from the Badger database during the `FinalizeBlock` or `Commit` methods. +This is not an accident. If the application received an error from the database, there +is no deterministic way for it to make progress so the only safe option is to terminate. + +### 1.3.4 Query + +When a client tries to read some information from the `kvstore`, the request will be +handled in the `Query` method. To do this, let's rewrite the `Query` method in `app.go`: + +```go +func (app *KVStoreApplication) Query(_ context.Context, req *abcitypes.RequestQuery) (*abcitypes.ResponseQuery, error) { + resp := abcitypes.ResponseQuery{Key: req.Data} + + dbErr := app.db.View(func(txn *badger.Txn) error { + item, err := txn.Get(req.Data) + if err != nil { + if err != badger.ErrKeyNotFound { + return err + } + resp.Log = "key does not exist" + return nil + } + + return item.Value(func(val []byte) error { + resp.Log = "exists" + resp.Value = val + return nil + }) + }) + if dbErr != nil { + log.Panicf("Error reading database, unable to execute query: %v", dbErr) + } + return &resp, nil +} +``` + +Since it reads only committed data from the store, transactions that are part of a block +that is being processed are not reflected in the query result. + +### 1.3.5 PrepareProposal and ProcessProposal + +`PrepareProposal` and `ProcessProposal` are methods introduced in CometBFT v0.37.0 +to give the application more control over the construction and processing of transaction blocks. + +When CometBFT sees that valid transactions (validated through `CheckTx`) are available to be +included in blocks, it groups some of these transactions and then gives the application a chance +to modify the group by invoking `PrepareProposal`. + +The application is free to modify the group before returning from the call, as long as the resulting set +does not use more bytes than `RequestPrepareProposal.max_tx_bytes`. +For example, the application may reorder, add, or even remove transactions from the group to improve the +execution of the block once accepted. + +In the following code, the application simply returns the unmodified group of transactions: + +```go +func (app *KVStoreApplication) PrepareProposal(_ context.Context, proposal *abcitypes.RequestPrepareProposal) (*abcitypes.ResponsePrepareProposal, error) { + return &abcitypes.ResponsePrepareProposal{Txs: proposal.Txs}, nil +} +``` + +Once a proposed block is received by a node, the proposal is passed to the +application to determine its validity before voting to accept the proposal. + +This mechanism may be used for different reasons, for example to deal with blocks manipulated +by malicious nodes, in which case the block should not be considered valid. + +The following code simply accepts all proposals: + +```go +func (app *KVStoreApplication) ProcessProposal(_ context.Context, proposal *abcitypes.RequestProcessProposal) (*abcitypes.ResponseProcessProposal, error) { + return &abcitypes.ResponseProcessProposal{Status: abcitypes.ResponseProcessProposal_ACCEPT}, nil +} +``` + +## 1.4 Starting an application and a CometBFT instance + +Now that we have the basic functionality of our application in place, let's put +it all together inside of our `main.go` file. + +Change the contents of your `main.go` file to the following. + +```go +package main + +import ( + "flag" + "fmt" + abciserver "github.com/cometbft/cometbft/abci/server" + "log" + "os" + "os/signal" + "path/filepath" + "syscall" + + "github.com/dgraph-io/badger/v3" + cmtlog "github.com/cometbft/cometbft/libs/log" +) + +var homeDir string +var socketAddr string + +func init() { + flag.StringVar(&homeDir, "kv-home", "", "Path to the kvstore directory (if empty, uses $HOME/.kvstore)") + flag.StringVar(&socketAddr, "socket-addr", "unix://example.sock", "Unix domain socket address (if empty, uses \"unix://example.sock\"") +} + +func main() { + flag.Parse() + if homeDir == "" { + homeDir = os.ExpandEnv("$HOME/.kvstore") + } + + dbPath := filepath.Join(homeDir, "badger") + db, err := badger.Open(badger.DefaultOptions(dbPath)) + if err != nil { + log.Fatalf("Opening database: %v", err) + } + + defer func() { + if err := db.Close(); err != nil { + log.Fatalf("Closing database: %v", err) + } + }() + + app := NewKVStoreApplication(db) + logger := cmtlog.NewTMLogger(cmtlog.NewSyncWriter(os.Stdout)) + + server := abciserver.NewSocketServer(socketAddr, app) + server.SetLogger(logger) + + if err := server.Start(); err != nil { + fmt.Fprintf(os.Stderr, "error starting socket server: %v", err) + + os.Exit(1) + } + defer server.Stop() + + c := make(chan os.Signal, 1) + signal.Notify(c, os.Interrupt, syscall.SIGTERM) + <-c +} +``` + +This is a huge blob of code, so let's break it down into pieces. + +First, we initialize the Badger database and create an app instance: + +```go +dbPath := filepath.Join(homeDir, "badger") +db, err := badger.Open(badger.DefaultOptions(dbPath)) +if err != nil { + log.Fatalf("Opening database: %v", err) +} +defer func() { + if err := db.Close(); err != nil { + log.Fatalf("Closing database: %v", err) + } +}() + +app := NewKVStoreApplication(db) +``` + +Then we start the ABCI server and add some signal handling to gracefully stop +it upon receiving SIGTERM or Ctrl-C. CometBFT will act as a client, +which connects to our server and send us transactions and other messages. + +```go +server := abciserver.NewSocketServer(socketAddr, app) +server.SetLogger(logger) + +if err := server.Start(); err != nil { + fmt.Fprintf(os.Stderr, "error starting socket server: %v", err) + os.Exit(1) +} +defer server.Stop() + +c := make(chan os.Signal, 1) +signal.Notify(c, os.Interrupt, syscall.SIGTERM) +<-c +``` + +## 1.5 Initializing and Running + +Our application is almost ready to run, but first we'll need to populate the CometBFT configuration files. +The following command will create a `cometbft-home` directory in your project and add a basic set of configuration files in `cometbft-home/config/`. +For more information on what these files contain see [the configuration documentation](https://github.com/cometbft/cometbft/blob/v0.38.x/docs/core/configuration.md). + +From the root of your project, run: + +```bash +go run github.com/cometbft/cometbft/cmd/cometbft@v0.38.0 init --home /tmp/cometbft-home +``` + +You should see an output similar to the following: + +```bash +I[2023-04-25|09:06:34.444] Generated private validator module=main keyFile=/tmp/cometbft-home/config/priv_validator_key.json stateFile=/tmp/cometbft-home/data/priv_validator_state.json +I[2023-04-25|09:06:34.444] Generated node key module=main path=/tmp/cometbft-home/config/node_key.json +I[2023-04-25|09:06:34.444] Generated genesis file module=main path=/tmp/cometbft-home/config/genesis.json +``` + +Now rebuild the app: + +```bash +go build -mod=mod # use -mod=mod to automatically refresh the dependencies +``` + +Everything is now in place to run your application. Run: + +```bash +./kvstore -kv-home /tmp/badger-home +``` + +The application will start and you should see an output similar to the following: + +```bash +badger 2023-04-25 17:01:28 INFO: All 0 tables opened in 0s +badger 2023-04-25 17:01:28 INFO: Discard stats nextEmptySlot: 0 +badger 2023-04-25 17:01:28 INFO: Set nextTxnTs to 0 +I[2023-04-25|17:01:28.726] service start msg="Starting ABCIServer service" impl=ABCIServer +I[2023-04-25|17:01:28.726] Waiting for new connection... +``` + +Then we need to start CometBFT service and point it to our application. +Open a new terminal window and cd to the same folder where the app is running. +Then execute the following command: + +```bash +go run github.com/cometbft/cometbft/cmd/cometbft@v0.38.0 node --home /tmp/cometbft-home --proxy_app=unix://example.sock +``` + +This should start the full node and connect to our ABCI application, which will be +reflected in the application output. + +```sh +I[2023-04-25|17:07:08.124] service start msg="Starting ABCIServer service" impl=ABCIServer +I[2023-04-25|17:07:08.124] Waiting for new connection... +I[2023-04-25|17:08:12.702] Accepted a new connection +I[2023-04-25|17:08:12.703] Waiting for new connection... +I[2023-04-25|17:08:12.703] Accepted a new connection +I[2023-04-25|17:08:12.703] Waiting for new connection... +``` + +Also, the application using CometBFT Core is producing blocks 🎉🎉 and you can see this reflected in the log output of the service in lines like this: + +```bash +I[2023-04-25|09:08:52.147] received proposal module=consensus proposal="Proposal{2/0 (F518444C0E348270436A73FD0F0B9DFEA758286BEB29482F1E3BEA75330E825C:1:C73D3D1273F2, -1) AD19AE292A45 @ 2023-04-25T12:08:52.143393Z}" +I[2023-04-25|09:08:52.147] received proposal module=consensus proposal="Proposal{2/0 (F518444C0E348270436A73FD0F0B9DFEA758286BEB29482F1E3BEA75330E825C:1:C73D3D1273F2, -1) AD19AE292A45 @ 2023-04-25T12:08:52.143393Z}" +I[2023-04-25|09:08:52.152] received complete proposal block module=consensus height=2 hash=F518444C0E348270436A73FD0F0B9DFEA758286BEB29482F1E3BEA75330E825C +I[2023-04-25|09:08:52.160] finalizing commit of block module=consensus height=2 hash=F518444C0E348270436A73FD0F0B9DFEA758286BEB29482F1E3BEA75330E825C root= num_txs=0 +I[2023-04-25|09:08:52.167] executed block module=state height=2 num_valid_txs=0 num_invalid_txs=0 +I[2023-04-25|09:08:52.171] committed state module=state height=2 num_txs=0 app_hash= +``` + +The blocks, as you can see from the `num_valid_txs=0` part, are empty, but let's remedy that next. + +## 1.6 Using the application + +Let's try submitting a transaction to our new application. +Open another terminal window and run the following curl command: + +```bash +curl -s 'localhost:26657/broadcast_tx_commit?tx="cometbft=rocks"' +``` + +If everything went well, you should see a response indicating which height the +transaction was included in the blockchain. + +Finally, let's make sure that transaction really was persisted by the application. +Run the following command: + +```bash +curl -s 'localhost:26657/abci_query?data="cometbft"' +``` + +Let's examine the response object that this request returns. +The request returns a `json` object with a `key` and `value` field set. + +```json +... + "key": "dGVuZGVybWludA==", + "value": "cm9ja3M=", +... +``` + +Those values don't look like the `key` and `value` we sent to CometBFT. +What's going on here? + +The response contains a `base64` encoded representation of the data we submitted. +To get the original value out of this data, we can use the `base64` command line utility: + +```bash +echo "cm9ja3M=" | base64 -d +``` + +## Outro + +Hope you could run everything smoothly. If you have any difficulties running through this tutorial, reach out to us via [discord](https://discord.com/invite/interchain) or open a new [issue](https://github.com/cometbft/cometbft/issues/new/choose) on Github. diff --git a/cometbft/v0.38/docs/guides/Install-CometBFT.mdx b/cometbft/v0.38/docs/guides/Install-CometBFT.mdx new file mode 100644 index 00000000..c7eaff20 --- /dev/null +++ b/cometbft/v0.38/docs/guides/Install-CometBFT.mdx @@ -0,0 +1,125 @@ +--- +order: 3 +--- + +# Install CometBFT + +## From Go Package + +Install the latest version of CometBFT's Go package: + +```sh +go install github.com/cometbft/cometbft/cmd/cometbft@latest +``` + +Install a specific version of CometBFT's Go package: + +```sh +go install github.com/cometbft/cometbft/cmd/cometbft@v0.38 +``` + +## From Binary + +To download pre-built binaries, see the [releases page](https://github.com/cometbft/cometbft/releases). + +## From Source + +You'll need `go` [installed](https://golang.org/doc/install) and the required +environment variables set, which can be done with the following commands: + +```sh +echo export GOPATH=\"\$HOME/go\" >> ~/.bash_profile +echo export PATH=\"\$PATH:\$GOPATH/bin\" >> ~/.bash_profile +``` + +### Get Source Code + +```sh +git clone https://github.com/cometbft/cometbft.git +cd cometbft +``` + +### Compile + +```sh +make install +``` + +to put the binary in `$GOPATH/bin` or use: + +```sh +make build +``` + +to put the binary in `./build`. + +_DISCLAIMER_ The binary of CometBFT is build/installed without the DWARF +symbol table. If you would like to build/install CometBFT with the DWARF +symbol and debug information, remove `-s -w` from `BUILD_FLAGS` in the make +file. + +The latest CometBFT is now installed. You can verify the installation by +running: + +```sh +cometbft version +``` + +## Reinstall + +If you already have CometBFT installed, and you make updates, simply + +```sh +make install +``` + +To upgrade, run + +```sh +git pull origin main +make install +``` + +## Compile with CLevelDB support + +Install [LevelDB](https://github.com/google/leveldb) (minimum version is 1.7). + +Install LevelDB with snappy (optionally). Below are commands for Ubuntu: + +```sh +sudo apt-get update +sudo apt install build-essential + +sudo apt-get install libsnappy-dev + +wget https://github.com/google/leveldb/archive/v1.23.tar.gz && \ + tar -zxvf v1.23.tar.gz && \ + cd leveldb-1.23/ && \ + make && \ + sudo cp -r out-static/lib* out-shared/lib* /usr/local/lib/ && \ + cd include/ && \ + sudo cp -r leveldb /usr/local/include/ && \ + sudo ldconfig && \ + rm -f v1.23.tar.gz +``` + +Set a database backend to `cleveldb`: + +```toml +# config/config.toml +db_backend = "cleveldb" +``` + +To install CometBFT, run: + +```sh +CGO_LDFLAGS="-lsnappy" make install COMETBFT_BUILD_OPTIONS=cleveldb +``` + +or run: + +```sh +CGO_LDFLAGS="-lsnappy" make build COMETBFT_BUILD_OPTIONS=cleveldb +``` + +which puts the binary in `./build`. diff --git a/cometbft/v0.38/docs/guides/Quick-Start.mdx b/cometbft/v0.38/docs/guides/Quick-Start.mdx new file mode 100644 index 00000000..d5f14817 --- /dev/null +++ b/cometbft/v0.38/docs/guides/Quick-Start.mdx @@ -0,0 +1,164 @@ +--- +order: 2 +--- + +# Quick Start + +## Overview + +This is a quick start guide. If you have a vague idea about how CometBFT +works and want to get started right away, continue. + +## Install + +See the [install guide](./install.md). + +## Initialization + +Running: + +```sh +cometbft init +``` + +will create the required files for a single, local node. + +These files are found in `$HOME/.cometbft`: + +```sh +$ ls $HOME/.cometbft + +config data + +$ ls $HOME/.cometbft/config/ + +config.toml genesis.json node_key.json priv_validator.json +``` + +For a single, local node, no further configuration is required. +Configuring a cluster is covered further below. + +## Local Node + +Start CometBFT with a simple in-process application: + +```sh +cometbft node --proxy_app=kvstore +``` + +> Note: `kvstore` is a non persistent app, if you would like to run an application with persistence run `--proxy_app=persistent_kvstore` + +and blocks will start to stream in: + +```sh +I[01-06|01:45:15.592] Executed block module=state height=1 validTxs=0 invalidTxs=0 +I[01-06|01:45:15.624] Committed state module=state height=1 txs=0 appHash= +``` + +Check the status with: + +```sh +curl -s localhost:26657/status +``` + +### Sending Transactions + +With the KVstore app running, we can send transactions: + +```sh +curl -s 'localhost:26657/broadcast_tx_commit?tx="abcd"' +``` + +and check that it worked with: + +```sh +curl -s 'localhost:26657/abci_query?data="abcd"' +``` + +We can send transactions with a key and value too: + +```sh +curl -s 'localhost:26657/broadcast_tx_commit?tx="name=satoshi"' +``` + +and query the key: + +```sh +curl -s 'localhost:26657/abci_query?data="name"' +``` + +where the value is returned in hex. + +## Cluster of Nodes + +First create four Ubuntu cloud machines. The following was tested on Digital +Ocean Ubuntu 16.04 x64 (3GB/1CPU, 20GB SSD). We'll refer to their respective IP +addresses below as IP1, IP2, IP3, IP4. + +Then, `ssh` into each machine and install CometBFT following the [instructions](./install.md). + +Next, use the `cometbft testnet` command to create four directories of config files (found in `./mytestnet`) and copy each directory to the relevant machine in the cloud, so that each machine has `$HOME/mytestnet/node[0-3]` directory. + +Before you can start the network, you'll need peers identifiers (IPs are not enough and can change). We'll refer to them as ID1, ID2, ID3, ID4. + +```sh +cometbft show_node_id --home ./mytestnet/node0 +cometbft show_node_id --home ./mytestnet/node1 +cometbft show_node_id --home ./mytestnet/node2 +cometbft show_node_id --home ./mytestnet/node3 +``` + +Here's a handy Bash script to compile the persistent peers string, which will +be needed for our next step: + +```bash +#!/bin/bash + +# Check if the required argument is provided +if [ $# -eq 0 ]; then + echo "Usage: $0 ..." + exit 1 +fi + +# Command to run on each IP +BASE_COMMAND="cometbft show_node_id --home ./mytestnet/node" + +# Initialize an array to store results +PERSISTENT_PEERS="" + +# Iterate through provided IPs +for i in "${!@}"; do + IP="${!i}" + NODE_IDX=$((i - 1)) # Adjust for zero-based indexing + + echo "Getting ID of $IP (node $NODE_IDX)..." + + # Run the command on the current IP and capture the result + ID=$($BASE_COMMAND$NODE_IDX) + + # Store the result in the array + PERSISTENT_PEERS+="$ID@$IP:26656" + + # Add a comma if not the last IP + if [ $i -lt $# ]; then + PERSISTENT_PEERS+="," + fi +done + +echo "$PERSISTENT_PEERS" +``` + +Finally, from each machine, run: + +```sh +cometbft node --home ./mytestnet/node0 --proxy_app=kvstore --p2p.persistent_peers="ID1@IP1:26656,ID2@IP2:26656,ID3@IP3:26656,ID4@IP4:26656" +cometbft node --home ./mytestnet/node1 --proxy_app=kvstore --p2p.persistent_peers="ID1@IP1:26656,ID2@IP2:26656,ID3@IP3:26656,ID4@IP4:26656" +cometbft node --home ./mytestnet/node2 --proxy_app=kvstore --p2p.persistent_peers="ID1@IP1:26656,ID2@IP2:26656,ID3@IP3:26656,ID4@IP4:26656" +cometbft node --home ./mytestnet/node3 --proxy_app=kvstore --p2p.persistent_peers="ID1@IP1:26656,ID2@IP2:26656,ID3@IP3:26656,ID4@IP4:26656" +``` + +Note that after the third node is started, blocks will start to stream in +because >2/3 of validators (defined in the `genesis.json`) have come online. +Persistent peers can also be specified in the `config.toml`. See [here](../core/configuration.md) for more information about configuration options. + +Transactions can then be sent as covered in the single, local node example above. diff --git a/cometbft/v0.38/docs/guides/README.md b/cometbft/v0.38/docs/guides/README.md new file mode 100644 index 00000000..b5633963 --- /dev/null +++ b/cometbft/v0.38/docs/guides/README.md @@ -0,0 +1,12 @@ +--- +order: false +parent: + order: 2 +--- + +# Guides + +- [Installing CometBFT](./install.md) +- [Quick-start using CometBFT](./quick-start.md) +- [Creating a built-in application in Go](./go-built-in.md) +- [Creating an external application in Go](./go.md) diff --git a/cometbft/v0.38/docs/imgs/abci.png b/cometbft/v0.38/docs/imgs/abci.png new file mode 100644 index 00000000..73111caf Binary files /dev/null and b/cometbft/v0.38/docs/imgs/abci.png differ diff --git a/cometbft/v0.38/docs/imgs/consensus_logic.png b/cometbft/v0.38/docs/imgs/consensus_logic.png new file mode 100644 index 00000000..22b70b26 Binary files /dev/null and b/cometbft/v0.38/docs/imgs/consensus_logic.png differ diff --git a/cometbft/v0.38/docs/imgs/contributing.png b/cometbft/v0.38/docs/imgs/contributing.png new file mode 100644 index 00000000..bb4bc6b5 Binary files /dev/null and b/cometbft/v0.38/docs/imgs/contributing.png differ diff --git a/cometbft/v0.38/docs/imgs/light_client_bisection_alg.png b/cometbft/v0.38/docs/imgs/light_client_bisection_alg.png new file mode 100644 index 00000000..a960ee69 Binary files /dev/null and b/cometbft/v0.38/docs/imgs/light_client_bisection_alg.png differ diff --git a/cometbft/v0.38/docs/imgs/sentry_layout.png b/cometbft/v0.38/docs/imgs/sentry_layout.png new file mode 100644 index 00000000..7d7dff44 Binary files /dev/null and b/cometbft/v0.38/docs/imgs/sentry_layout.png differ diff --git a/cometbft/v0.38/docs/imgs/sentry_local_config.png b/cometbft/v0.38/docs/imgs/sentry_local_config.png new file mode 100644 index 00000000..4fdb2fe5 Binary files /dev/null and b/cometbft/v0.38/docs/imgs/sentry_local_config.png differ diff --git a/cometbft/v0.38/docs/introduction/intro.mdx b/cometbft/v0.38/docs/introduction/intro.mdx new file mode 100644 index 00000000..964e6036 --- /dev/null +++ b/cometbft/v0.38/docs/introduction/intro.mdx @@ -0,0 +1,333 @@ +--- +order: 1 +parent: + title: Introduction + order: 1 +--- +{/* trigger rebuild */} +{/* trigger rebuild */} + + +# What is CometBFT + +CometBFT is software for securely and consistently replicating an +application on many machines. By securely, we mean that CometBFT works +as long as less than 1/3 of machines fail in arbitrary ways. By consistently, +we mean that every non-faulty machine sees the same transaction log and +computes the same state. Secure and consistent replication is a +fundamental problem in distributed systems; it plays a critical role in +the fault tolerance of a broad range of applications, from currencies, +to elections, to infrastructure orchestration, and beyond. + +The ability to tolerate machines failing in arbitrary ways, including +becoming malicious, is known as Byzantine fault tolerance (BFT). The +theory of BFT is decades old, but software implementations have only +became popular recently, due largely to the success of "blockchain +technology" like Bitcoin and Ethereum. Blockchain technology is just a +reformalization of BFT in a more modern setting, with emphasis on +peer-to-peer networking and cryptographic authentication. The name +derives from the way transactions are batched in blocks, where each +block contains a cryptographic hash of the previous one, forming a +chain. + +CometBFT consists of two chief technical components: a blockchain +consensus engine and a generic application interface. +The consensus engine, +which is based on [Tendermint consensus algorithm][tendermint-paper], +ensures that the same transactions are +recorded on every machine in the same order. The application interface, +called the Application BlockChain Interface (ABCI), delivers the transactions +to applications for processing. Unlike other +blockchain and consensus solutions, which come pre-packaged with built +in state machines (like a fancy key-value store, or a quirky scripting +language), developers can use CometBFT for BFT state machine +replication of applications written in whatever programming language and +development environment is right for them. + +CometBFT is designed to be easy-to-use, simple-to-understand, highly +performant, and useful for a wide variety of distributed applications. + +## CometBFT vs. X + +CometBFT is broadly similar to two classes of software. The first +class consists of distributed key-value stores, like Zookeeper, etcd, +and consul, which use non-BFT consensus. The second class is known as +"blockchain technology", and consists of both cryptocurrencies like +Bitcoin and Ethereum, and alternative distributed ledger designs like +Hyperledger's Burrow. + +### Zookeeper, etcd, consul + +Zookeeper, etcd, and consul are all implementations of key-value stores +atop a classical, non-BFT consensus algorithm. Zookeeper uses an +algorithm called Zookeeper Atomic Broadcast, while etcd and consul use +the Raft log replication algorithm. A +typical cluster contains 3-5 machines, and can tolerate crash failures +in less than 1/2 of the machines (e.g., 1 out of 3 or 2 out of 5), +but even a single Byzantine fault can jeopardize the whole system. + +Each offering provides a slightly different implementation of a +featureful key-value store, but all are generally focused around +providing basic services to distributed systems, such as dynamic +configuration, service discovery, locking, leader-election, and so on. + +CometBFT is in essence similar software, but with two key differences: + +- It is Byzantine Fault Tolerant, meaning it can only tolerate less than 1/3 + of machines failing, but those failures can include arbitrary behavior - + including hacking and malicious attacks. +- It does not specify a + particular application, like a fancy key-value store. Instead, it + focuses on arbitrary state machine replication, so developers can build + the application logic that's right for them, from key-value store to + cryptocurrency to e-voting platform and beyond. + +### Bitcoin, Ethereum, etc + +[Tendermint consensus algorithm][tendermint-paper], adopted by CometBFT, +emerged in the tradition of cryptocurrencies like Bitcoin, +Ethereum, etc. with the goal of providing a more efficient and secure +consensus algorithm than Bitcoin's Proof of Work. In the early days, +Tendermint consensus-based blockchains had a simple currency built in, and to participate in +consensus, users had to "bond" units of the currency into a security +deposit which could be revoked if they misbehaved -this is what made +Tendermint consensus a Proof-of-Stake algorithm. + +Since then, CometBFT has evolved to be a general purpose blockchain +consensus engine that can host arbitrary application states. That means +it can be used as a plug-and-play replacement for the consensus engines +of other blockchain software. So one can take the current Ethereum code +base, whether in Rust, or Go, or Haskell, and run it as an ABCI +application using CometBFT. Indeed, [we did that with +Ethereum](https://github.com/cosmos/ethermint). And we plan to do +the same for Bitcoin, ZCash, and various other deterministic +applications as well. + +Another example of a cryptocurrency application built on CometBFT is +[the Cosmos network](http://cosmos.network). + +### Other Blockchain Projects + +[Fabric](https://github.com/hyperledger/fabric) takes a similar approach +to CometBFT, but is more opinionated about how the state is managed, +and requires that all application behavior runs in potentially many +docker containers, modules it calls "chaincode". It uses an +implementation of [PBFT](http://pmg.csail.mit.edu/papers/osdi99.pdf). +from a team at IBM that is [augmented to handle potentially +non-deterministic +chaincode](https://drops.dagstuhl.de/opus/volltexte/2017/7093/pdf/LIPIcs-OPODIS-2016-24.pdf). +It is possible to implement this docker-based behavior as an ABCI app in +CometBFT, though extending CometBFT to handle non-determinism +remains for future work. + +[Burrow](https://github.com/hyperledger/burrow) is an implementation of +the Ethereum Virtual Machine and Ethereum transaction mechanics, with +additional features for a name-registry, permissions, and native +contracts, and an alternative blockchain API. It uses CometBFT as its +consensus engine, and provides a particular application state. + +## ABCI Overview + +The [Application BlockChain Interface +(ABCI)](https://github.com/cometbft/cometbft/tree/v0.38.x/abci) +allows for Byzantine Fault Tolerant replication of applications +written in any programming language. + +### Motivation + +Thus far, all blockchains "stacks" (such as +[Bitcoin](https://github.com/bitcoin/bitcoin)) have had a monolithic +design. That is, each blockchain stack is a single program that handles +all the concerns of a decentralized ledger; this includes P2P +connectivity, the "mempool" broadcasting of transactions, consensus on +the most recent block, account balances, Turing-complete contracts, +user-level permissions, etc. + +Using a monolithic architecture is typically bad practice in computer +science. It makes it difficult to reuse components of the code, and +attempts to do so result in complex maintenance procedures for forks of +the codebase. This is especially true when the codebase is not modular +in design and suffers from "spaghetti code". + +Another problem with monolithic design is that it limits you to the +language of the blockchain stack (or vice versa). In the case of +Ethereum which supports a Turing-complete bytecode virtual-machine, it +limits you to languages that compile down to that bytecode; while the +[list](https://github.com/pirapira/awesome-ethereum-virtual-machine#programming-languages-that-compile-into-evm) +is growing, it is still very limited. + +In contrast, our approach is to decouple the consensus engine and P2P +layers from the details of the state of the particular +blockchain application. We do this by abstracting away the details of +the application to an interface, which is implemented as a socket +protocol. + +### Intro to ABCI + +[CometBFT](https://github.com/cometbft/cometbft), the +"consensus engine", communicates with the application via a socket +protocol that satisfies the ABCI, the CometBFT Socket Protocol. + +To draw an analogy, let's talk about a well-known cryptocurrency, +Bitcoin. Bitcoin is a cryptocurrency blockchain where each node +maintains a fully audited Unspent Transaction Output (UTXO) database. If +one wanted to create a Bitcoin-like system on top of ABCI, CometBFT +would be responsible for + +- Sharing blocks and transactions between nodes +- Establishing a canonical/immutable order of transactions + (the blockchain) + +The application will be responsible for + +- Maintaining the UTXO database +- Validating cryptographic signatures of transactions +- Preventing transactions from spending non-existent transactions +- Allowing clients to query the UTXO database. + +CometBFT is able to decompose the blockchain design by offering a very +simple API (i.e. the ABCI) between the application process and consensus +process. + +The ABCI consists of 3 primary message types that get delivered from the +core to the application. The application replies with corresponding +response messages. + +The messages are specified here: [ABCI Message +Types](https://github.com/cometbft/cometbft/blob/v0.38.x/proto/tendermint/abci/types.proto). + +The **FinalizeBlock** message is the work horse of the application. Each +transaction in the blockchain is finalized within this message. The +application needs to validate each transaction received with the +**FinalizeBlock** message against the current state, application protocol, +and the cryptographic credentials of the transaction. FinalizeBlock only +prepares the update to be made and does not change the state of the application. +The state change is actually committed in a later stage i.e. in commit phase. + +The **CheckTx** message is used for validating transactions. +CometBFT's mempool first checks the +validity of a transaction with **CheckTx**, and only relays valid +transactions to its peers. For instance, an application may check an +incrementing sequence number in the transaction and return an error upon +**CheckTx** if the sequence number is old. Alternatively, they might use +a capabilities based system that requires capabilities to be renewed +with every transaction. + +The **Commit** message is used to compute a cryptographic commitment to +the current application state, to be placed into the next block header. +This has some handy properties. Inconsistencies in updating that state +will now appear as blockchain forks which catches a whole class of +programming errors. This also simplifies the development of secure +lightweight clients, as Merkle-hash proofs can be verified by checking +against the block hash, and that the block hash is signed by a quorum. + +There can be multiple ABCI socket connections to an application. +CometBFT creates four ABCI connections to the application; one +for the validation of transactions when broadcasting in the mempool, one for +the consensus engine to run block proposals, one for creating snapshots of the +application state, and one more for querying the application state. + +It's probably evident that application designers need to very carefully +design their message handlers to create a blockchain that does anything +useful but this architecture provides a place to start. The diagram +below illustrates the flow of messages via ABCI. + +![abci](../imgs/abci.png) + +## A Note on Determinism + +The logic for blockchain transaction processing must be deterministic. +If the application logic weren't deterministic, consensus would not be +reached among the CometBFT replica nodes. + +Solidity on Ethereum is a great language of choice for blockchain +applications because, among other reasons, it is a completely +deterministic programming language. However, it's also possible to +create deterministic applications using existing popular languages like +Java, C++, Python, or Go, by avoiding +sources of non-determinism such as: + +- random number generators (without deterministic seeding) +- race conditions on threads (or avoiding threads altogether) +- the system clock +- uninitialized memory (in unsafe programming languages like C + or C++) +- [floating point + arithmetic](http://gafferongames.com/networking-for-game-programmers/floating-point-determinism/) +- language features that are random (e.g. map iteration in Go) + +While programmers can avoid non-determinism by being careful, it is also +possible to create a special linter or static analyzer for each language +to check for determinism. In the future we may work with partners to +create such tools. + +## Consensus Overview + +CometBFT adopts [Tendermint consensus][tendermint-paper], +an easy-to-understand, mostly asynchronous, BFT consensus algorithm. +The algorithm follows a simple state machine that looks like this: + +![consensus-logic](../imgs/consensus_logic.png) + +Participants in the algorithm are called **validators**; they take turns +proposing blocks of transactions and voting on them. Blocks are +committed in a chain, with one block at each **height**. A block may +fail to be committed, in which case the algorithm moves to the next +**round**, and a new validator gets to propose a block for that height. +Two stages of voting are required to successfully commit a block; we +call them **pre-vote** and **pre-commit**. + +There is a picture of a couple doing the polka because validators are +doing something like a polka dance. When more than two-thirds of the +validators pre-vote for the same block, we call that a **polka**. Every +pre-commit must be justified by a polka in the same round. +A block is committed when +more than 2/3 of validators pre-commit for the same block in the same +round. + +Validators may fail to commit a block for a number of reasons; the +current proposer may be offline, or the network may be slow. Tendermint consensus +allows them to establish that a validator should be skipped. Validators +wait a small amount of time to receive a complete proposal block from +the proposer before voting to move to the next round. This reliance on a +timeout is what makes Tendermint consensus a weakly synchronous algorithm, rather +than an asynchronous one. However, the rest of the algorithm is +asynchronous, and validators only make progress after hearing from more +than two-thirds of the validator set. A simplifying element of +Tendermint consensus is that it uses the same mechanism to commit a block as it +does to skip to the next round. + +Assuming less than one-third of the validators are Byzantine, Tendermint consensus algorithm +guarantees that safety will never be violated - that is, validators will +never commit conflicting blocks at the same height. To do this it +introduces a few **locking** rules which modulate which paths can be +followed in the flow diagram. Once a validator precommits a block, it is +locked on that block. Then, + +1. it must prevote for the block it is locked on +2. it can only unlock, and precommit for a new block, if there is a + polka for that block in a later round + +## Stake + +In many systems, not all validators will have the same "weight" in the +consensus protocol. Thus, we are not so much interested in one-third or +two-thirds of the validators, but in those proportions of the total +voting power, which may not be uniformly distributed across individual +validators. + +Since CometBFT can replicate arbitrary applications, it is possible to +define a currency, and denominate the voting power in that currency. +When voting power is denominated in a native currency, the system is +often referred to as Proof-of-Stake. Validators can be forced, by logic +in the application, to "bond" their currency holdings in a security +deposit that can be destroyed if they're found to misbehave in the +consensus protocol. This adds an economic element to the security of the +protocol, allowing one to quantify the cost of violating the assumption +that less than one-third of voting power is Byzantine. + +The [Cosmos Network](https://cosmos.network) is designed to use this +Proof-of-Stake mechanism across an array of cryptocurrencies implemented +as ABCI applications. + +[tendermint-paper]: https://arxiv.org/abs/1807.04938 diff --git a/cometbft/v0.38/docs/networks/Docker-Compose.mdx b/cometbft/v0.38/docs/networks/Docker-Compose.mdx new file mode 100644 index 00000000..08aa4fce --- /dev/null +++ b/cometbft/v0.38/docs/networks/Docker-Compose.mdx @@ -0,0 +1,179 @@ +--- +order: 2 +--- + +# Docker Compose + +With Docker Compose, you can spin up local testnets with a single command. + +## Requirements + +1. [Install CometBFT](../guides/install.md) +2. [Install docker](https://docs.docker.com/engine/installation/) +3. [Install docker-compose](https://docs.docker.com/compose/install/) + +## Build + +Build the `cometbft` binary and, optionally, the `cometbft/localnode` +docker image. + +Note the binary will be mounted into the container so it can be updated without +rebuilding the image. + +```sh +# Build the linux binary in ./build +make build-linux + +# (optionally) Build cometbft/localnode image +make build-docker-localnode +``` + +## Run a testnet + +To start a 4 node testnet run: + +```sh +make localnet-start +``` + +The nodes bind their RPC servers to ports 26657, 26660, 26662, and 26664 on the +host. + +This file creates a 4-node network using the localnode image. + +The nodes of the network expose their P2P and RPC endpoints to the host machine +on ports 26656-26657, 26659-26660, 26661-26662, and 26663-26664 respectively. + +To update the binary, just rebuild it and restart the nodes: + +```sh +make build-linux +make localnet-start +``` + +## Configuration + +The `make localnet-start` creates files for a 4-node testnet in `./build` by +calling the `cometbft testnet` command. + +The `./build` directory is mounted to the `/cometbft` mount point to attach +the binary and config files to the container. + +To change the number of validators / non-validators change the `localnet-start` Makefile target [here](../../Makefile): + +```makefile +localnet-start: localnet-stop + @if ! [ -f build/node0/config/genesis.json ]; then docker run --rm -v $(CURDIR)/build:/cometbft:Z cometbft/localnode testnet --v 5 --n 3 --o . --populate-persistent-peers --starting-ip-address 192.167.10.2 ; fi + docker compose up -d +``` + +The command now will generate config files for 5 validators and 3 +non-validators. Along with generating new config files the docker-compose file needs to be edited. +Adding 4 more nodes is required in order to fully utilize the config files that were generated. + +```yml + node3: # bump by 1 for every node + container_name: node3 # bump by 1 for every node + image: "cometbft/localnode" + environment: + - ID=3 + - LOG=${LOG:-cometbft.log} + ports: + - "26663-26664:26656-26657" # Bump 26663-26664 by one for every node + volumes: + - ./build:/cometbft:Z + networks: + localnet: + ipv4_address: 192.167.10.5 # bump the final digit by 1 for every node +``` + +Before running it, don't forget to cleanup the old files: + +```sh +# Clear the build folder +rm -rf ./build/node* +``` + +## Configuring ABCI containers + +To use your own ABCI applications with 4-node setup edit the [docker-compose.yaml](https://github.com/cometbft/cometbft/blob/v0.38.x/docker-compose.yml) file and add images to your ABCI application. + +```yml + abci0: + container_name: abci0 + image: "abci-image" + build: + context: . + dockerfile: abci.Dockerfile + command: + networks: + localnet: + ipv4_address: 192.167.10.6 + + abci1: + container_name: abci1 + image: "abci-image" + build: + context: . + dockerfile: abci.Dockerfile + command: + networks: + localnet: + ipv4_address: 192.167.10.7 + + abci2: + container_name: abci2 + image: "abci-image" + build: + context: . + dockerfile: abci.Dockerfile + command: + networks: + localnet: + ipv4_address: 192.167.10.8 + + abci3: + container_name: abci3 + image: "abci-image" + build: + context: . + dockerfile: abci.Dockerfile + command: + networks: + localnet: + ipv4_address: 192.167.10.9 + +``` + +Override the [command](https://github.com/cometbft/cometbft/blob/v0.38.x/networks/local/localnode/Dockerfile#L11) in each node to connect to it's ABCI. + +```yml + node0: + container_name: node0 + image: "cometbft/localnode" + ports: + - "26656-26657:26656-26657" + environment: + - ID=0 + - LOG=$${LOG:-cometbft.log} + volumes: + - ./build:/cometbft:Z + command: node --proxy_app=tcp://abci0:26658 + networks: + localnet: + ipv4_address: 192.167.10.2 +``` + +Similarly do for node1, node2 and node3 then [run testnet](#run-a-testnet). + +## Logging + +Log is saved under the attached volume, in the `cometbft.log` file. If the +`LOG` environment variable is set to `stdout` at start, the log is not saved, +but printed on the screen. + +## Special binaries + +If you have multiple binaries with different names, you can specify which one +to run with the `BINARY` environment variable. The path of the binary is relative +to the attached volume. diff --git a/cometbft/v0.38/docs/networks/Overview.mdx b/cometbft/v0.38/docs/networks/Overview.mdx new file mode 100644 index 00000000..ceea2359 --- /dev/null +++ b/cometbft/v0.38/docs/networks/Overview.mdx @@ -0,0 +1,13 @@ +--- +order: 1 +parent: + title: Networks + order: 5 +--- + +# Overview + +Use [Docker Compose](./docker-compose.md) to spin up CometBFT testnets on your +local machine. + +See the `cometbft testnet --help` command for more help initializing testnets. diff --git a/cometbft/v0.38/docs/presubmit.sh b/cometbft/v0.38/docs/presubmit.sh new file mode 100755 index 00000000..0dec7d32 --- /dev/null +++ b/cometbft/v0.38/docs/presubmit.sh @@ -0,0 +1,39 @@ +#!/bin/bash +# +# This script verifies that each document in the docs and architecture +# directory has a corresponding table-of-contents entry in its README file. +# +# This can be run manually from the command line. +# It is also run in CI via the docs-toc.yml workflow. +# +set -euo pipefail + +readonly base="$(dirname $0)" +cd "$base" + +readonly workdir="$(mktemp -d)" +trap "rm -fr -- '$workdir'" EXIT + +checktoc() { + local dir="$1" + local tag="$2"'-*-*' + local out="$workdir/${dir}.out.txt" + ( + cd "$dir" >/dev/null + find . -maxdepth 1 -type f -name "$tag" -not -exec grep -q "({})" README.md ';' -print + ) > "$out" + if [[ -s "$out" ]] ; then + echo "-- The following files in $dir lack a ToC entry: +" + cat "$out" + return 1 + fi +} + +err=0 + +# Verify that each RFC and ADR has a ToC entry in its README file. +checktoc architecture adr || ((err++)) +checktoc rfc rfc || ((err++)) + +exit $err diff --git a/cometbft/v0.38/docs/qa/CometBFT-QA-34.mdx b/cometbft/v0.38/docs/qa/CometBFT-QA-34.mdx new file mode 100644 index 00000000..a7811987 --- /dev/null +++ b/cometbft/v0.38/docs/qa/CometBFT-QA-34.mdx @@ -0,0 +1,370 @@ +--- +order: 1 +parent: + title: CometBFT QA Results v0.34.x + description: This is a report on the results obtained when running v0.34.x on testnets + order: 3 +--- + +# CometBFT QA Results v0.34.x + +## v0.34.x - From Tendermint Core to CometBFT + +This section reports on the QA process we followed before releasing the first `v0.34.x` version +from our CometBFT repository. + +The changes with respect to the last version of `v0.34.x` +(namely `v0.34.26`, released from the Informal Systems' Tendermint Core fork) +are minimal, and focus on rebranding our fork of Tendermint Core to CometBFT at places +where there is no substantial risk of breaking compatibility +with earlier Tendermint Core versions of `v0.34.x`. + +Indeed, CometBFT versions of `v0.34.x` (`v0.34.27` and subsequent) should fulfill +the following compatibility-related requirements. + +* Operators can easily upgrade a `v0.34.x` version of Tendermint Core to CometBFT. +* Upgrades from Tendermint Core to CometBFT can be uncoordinated for versions of the `v0.34.x` branch. +* Nodes running CometBFT must be interoperable with those running Tendermint Core in the same chain, + as long as all are running a `v0.34.x` version. + +These QA tests focus on the third bullet, whereas the first two bullets are tested using our _e2e tests_. + +It would be prohibitively time consuming to test mixed networks of all combinations of existing `v0.34.x` +versions, combined with the CometBFT release candidate under test. +Therefore our testing focuses on the last Tendermint Core version (`v0.34.26`) and the CometBFT release +candidate under test. + +We run the _200 node test_, but not the _rotating node test_. The effort of running the latter +is not justified given the amount and nature of the changes we are testing with respect to the +full QA cycle run previously on `v0.34.x`. +Since the changes to the system's logic are minimal, we are interested in these performance requirements: + +* The CometBFT release candidate under test performs similarly to Tendermint Core (i.e., the baseline) + * when used at scale (i.e., in a large network of CometBFT nodes) + * when used at scale in a mixed network (i.e., some nodes are running CometBFT + and others are running an older Tendermint Core version) + +Therefore we carry out a complete run of the _200-node test_ on the following networks: + +* A homogeneous 200-node testnet, where all nodes are running the CometBFT release candidate under test. +* A mixed network where 1/2 (99 out of 200) of the nodes are running the CometBFT release candidate under test, + and the rest (101 out of 200) are running Tendermint Core `v0.34.26`. +* A mixed network where 1/3 (66 out of 200) of the nodes are running the CometBFT release candidate under test, + and the rest (134 out of 200) are running Tendermint Core `v0.34.26`. +* A mixed network where 2/3 (133 out of 200) of the nodes are running the CometBFT release candidate under test, + and the rest (67 out of 200) are running Tendermint Core `v0.34.26`. + +## Configuration and Results +In the following sections we provide the results of the _200 node test_. +Each section reports the baseline results (for reference), the homogeneous network scenario (all CometBFT nodes), +and the mixed networks with 1/2, 1/3 and 2/3 of Tendermint Core nodes. + +### Saturation Point + +As the CometBFT release candidate under test has minimal changes +with respect to Tendermint Core `v0.34.26`, other than the rebranding changes, +we can confidently reuse the results from the `v0.34.x` baseline test regarding +the [saturation point](TMCore-QA-34.md#finding-the-saturation-point). + +Therefore, we will simply use a load of (`r=200,c=2`) +(see the explanation [here](TMCore-QA-34.md#finding-the-saturation-point)) on all experiments. + +We also include the baseline results for quick reference and comparison. + +### Experiments + +On each of the three networks, the test consists of 4 experiments, with the goal of +ensuring the data obtained is consistent across experiments. + +On each of the networks, we pick only one representative run to present and discuss the +results. + + +## Examining latencies +For each network the figures plot the four experiments carried out with the network. +We can see that the latencies follow comparable patterns across all experiments. + +Unique identifiers, UUID, for each execution are presented on top of each graph. +We refer to these UUID to indicate to the representative runs. + +### CometBFT Homogeneous network + +![latencies](img34/homogeneous/all_experiments.png) + +### 1/2 Tendermint Core - 1/2 CometBFT + +![latencies](img34/cmt1tm1/all_experiments.png) + +### 1/3 Tendermint Core - 2/3 CometBFT + +![latencies](img34/cmt2tm1/all_experiments.png) + +### 2/3 Tendermint Core - 1/3 CometBFT + +![latencies_all_tm2_3_cmt1_3](img34/v034_200node_tm2cmt1/all_experiments.png) + + +## Prometheus Metrics + +This section reports on the key Prometheus metrics extracted from the following experiments. + +* Baseline results: `v0.34.x`, obtained in October 2022 and reported [here](TMCore-QA-34.md). +* CometBFT homogeneous network: experiment with UUID starting with `be8c`. +* Mixed network, 1/2 Tendermint Core `v0.34.26` and 1/2 running CometBFT: experiment with UUID starting with `04ee`. +* Mixed network, 1/3 Tendermint Core `v0.34.26` and 2/3 running CometBFT: experiment with UUID starting with `fc5e`. +* Mixed network, 2/3 Tendermint Core `v0.34.26` and 1/3 running CometBFT: experiment with UUID starting with `4759`. + +We make explicit comparisons between the baseline and the homogenous setups, but refrain from +commenting on the mixed network experiment unless they show some exceptional results. + +### Mempool Size + +For each reported experiment we show two graphs. +The first shows the evolution over time of the cumulative number of transactions +inside all full nodes' mempools at a given time. + +The second one shows the evolution of the average over all full nodes. + +#### Baseline + +![mempool-cumulative](img34/baseline/mempool_size.png) + +![mempool-avg](img34/baseline/avg_mempool_size.png) + +#### CometBFT Homogeneous network + +The results for the homogeneous network and the baseline are similar in terms of outstanding transactions. + +![mempool-cumulative-homogeneous](img34/homogeneous/mempool_size.png) + +![mempool-avg-homogeneous](img34/homogeneous/avg_mempool_size.png) + +#### 1/2 Tendermint Core - 1/2 CometBFT + +![mempool size](img34/cmt1tm1/mempool_size.png) + +![average mempool size](img34/cmt1tm1/avg_mempool_size.png) + +#### 1/3 Tendermint Core - 2/3 CometBFT + +![mempool size](img34/cmt2tm1/mempool_size.png) + +![average mempool size](img34/cmt2tm1/avg_mempool_size.png) + +#### 2/3 Tendermint Core - 1/3 CometBFT + +![mempool_tm2_3_cmt_1_3](img34/v034_200node_tm2cmt1/mempool_size.png) + +![mempool-avg_tm2_3_cmt_1_3](img34/v034_200node_tm2cmt1/avg_mempool_size.png) + +### Consensus Rounds per Height + +The following graphs show the rounds needed to complete each height and agree on a block. + +A value of `0` shows that only one round was required (with id `0`), and a value of `1` shows that two rounds were required. + +#### Baseline +We can see that round 1 is reached with a certain frequency. + +![rounds](img34/baseline/rounds.png) + +#### CometBFT Homogeneous network + +Most heights finished in round 0, some nodes needed to advance to round 1 at various moments, +and a few nodes even needed to advance to round 2 at one point. +This coincides with the time at which we observed the biggest peak in mempool size +on the corresponding plot, shown above. + +![rounds-homogeneous](img34/homogeneous/rounds.png) + +#### 1/2 Tendermint Core - 1/2 CometBFT + +![peers](img34/cmt1tm1/rounds.png) + +#### 1/3 Tendermint Core - 2/3 CometBFT + +![peers](img34/cmt2tm1/rounds.png) + +#### 2/3 Tendermint Core - 1/3 CometBFT + +![rounds-tm2_3_cmt1_3](img34/v034_200node_tm2cmt1/rounds.png) + +### Peers + +The following plots show how many peers a node had throughtout the experiment. + +The thick red dashed line represents the moving average over a sliding window of 20 seconds. + +#### Baseline + +The following graph shows the that the number of peers was stable throughout the experiment. +Seed nodes typically have a higher number of peers. +The fact that non-seed nodes reach more than 50 peers is due to +[#9548](https://github.com/tendermint/tendermint/issues/9548). + +![peers](img34/baseline/peers.png) + +#### CometBFT Homogeneous network + +The results for the homogeneous network are very similar to the baseline. +The only difference being that the seed nodes seem to loose peers in the middle of the experiment. +However this cannot be attributed to the differences in the code, which are mainly rebranding. + +![peers-homogeneous](img34/homogeneous/peers.png) + +#### 1/2 Tendermint Core - 1/2 CometBFT + +![peers](img34/cmt1tm1/peers.png) + +#### 1/3 Tendermint Core - 2/3 CometBFT + +![peers](img34/cmt2tm1/peers.png) + +#### 2/3 Tendermint Core - 1/3 CometBFT + +As in the homogeneous case, there is some variation in the number of peers for some nodes. +These, however, do not affect the average. + +![peers-tm2_3_cmt1_3](img34/v034_200node_tm2cmt1/peers.png) + +### Blocks Produced per Minute, Transactions Processed per Minute + +The following plot show the rate of block production and the rate of transactions delivered, throughout the experiments. + +In both graphs, rates are calculated over a sliding window of 20 seconds. +The thick red dashed line show the rates' moving averages. + +#### Baseline + +The average number of blocks/minute oscilate between 10 and 40. + +![heights](img34/baseline/block_rate_regular.png) + +The number of transactions/minute tops around 30k. + +![total-txs](img34/baseline/total_txs_rate_regular.png) + + +#### CometBFT Homogeneous network + +The plot showing the block production rate shows that the rate oscillates around 20 blocks/minute, +mostly within the same range as the baseline. + +![heights-homogeneous-rate](img34/homogeneous/block_rate_regular.png) + +The plot showing the transaction rate shows the rate stays around 20000 transactions per minute, +also topping around 30k. + +![txs-homogeneous-rate](img34/homogeneous/total_txs_rate_regular.png) + +#### 1/2 Tendermint Core - 1/2 CometBFT + +![height rate](img34/cmt1tm1/block_rate_regular.png) + +![transaction rate](img34/cmt1tm1/total_txs_rate_regular.png) + +#### 1/3 Tendermint Core - 2/3 CometBFT + +![height rate](img34/cmt2tm1/block_rate_regular.png) + +![transaction rate](img34/cmt2tm1/total_txs_rate_regular.png) + +#### 2/3 Tendermint Core - 1/3 CometBFT + +![height rate](img34/v034_200node_tm2cmt1/block_rate_regular.png) + +![transaction rate](img34/v034_200node_tm2cmt1/total_txs_rate_regular.png) + +### Memory Resident Set Size + +The following graphs show the Resident Set Size (RSS) of all monitored processes and the average value. + +#### Baseline + +![rss](img34/baseline/memory.png) + +![rss-avg](img34/baseline/avg_memory.png) + +#### CometBFT Homogeneous network + +This is the plot for the homogeneous network, which is slightly more stable than the baseline over +the time of the experiment. + +![rss-homogeneous](img34/homogeneous/memory.png) + +And this is the average plot. It oscillates around 560 MiB, which is noticeably lower than the baseline. + +![rss-avg-homogeneous](img34/homogeneous/avg_memory.png) + +#### 1/2 Tendermint Core - 1/2 CometBFT + +![rss](img34/cmt1tm1/memory.png) + +![rss average](img34/cmt1tm1/avg_memory.png) + +#### 1/3 Tendermint Core - 2/3 CometBFT + +![rss](img34/cmt2tm1/memory.png) + +![rss average](img34/cmt2tm1/avg_memory.png) + +#### 2/3 Tendermint Core - 1/3 CometBFT + +![rss](img34/v034_200node_tm2cmt1/memory.png) + +![rss average](img34/v034_200node_tm2cmt1/avg_memory.png) + +### CPU utilization + +The following graphs show the `load1` of nodes, as typically shown in the first line of the Unix `top` +command, and their average value. + +#### Baseline + +![load1](img34/baseline/cpu.png) + +![load1-avg](img34/baseline/avg_cpu.png) + +#### CometBFT Homogeneous network + +The load in the homogenous network is, similarly to the baseline case, below 5 and, therefore, normal. + +![load1-homogeneous](img34/homogeneous/cpu.png) + +As expected, the average plot also looks similar. + +![load1-homogeneous-avg](img34/homogeneous/avg_cpu.png) + +#### 1/2 Tendermint Core - 1/2 CometBFT + +![load1](img34/cmt1tm1/cpu.png) + +![average load1](img34/cmt1tm1/avg_cpu.png) + +#### 1/3 Tendermint Core - 2/3 CometBFT + +![load1](img34/cmt2tm1/cpu.png) + +![average load1](img34/cmt2tm1/avg_cpu.png) + +#### 2/3 Tendermint Core - 1/3 CometBFT + +![load1](img34/v034_200node_tm2cmt1/cpu.png) + +![average load1](img34/v034_200node_tm2cmt1/avg_cpu.png) + +## Test Results + +The comparison of the baseline results and the homogeneous case show that both scenarios had similar numbers and are therefore equivalent. + +The mixed nodes cases show that networks operate normally with a mix of compatible Tendermint Core and CometBFT versions. +Although not the main goal, a comparison of metric numbers with the homogenous case and the baseline scenarios show similar results and therefore we can conclude that mixing compatible Tendermint Core and CometBFT introduces not performance degradation. + +A conclusion of these tests is shown in the following table, along with the commit versions used in the experiments. + +| Scenario | Date | Version | Result | +|--|--|--|--| +| CometBFT Homogeneous network | 2023-02-08 | 3b783434f26b0e87994e6a77c5411927aad9ce3f | Pass | +| 1/2 Tendermint Core
1/2 CometBFT | 2023-02-14 | CometBFT: 3b783434f26b0e87994e6a77c5411927aad9ce3f
Tendermint Core: 66c2cb63416e66bff08e11f9088e21a0ed142790 | Pass | +| 1/3 Tendermint Core
2/3 CometBFT | 2023-02-08 | CometBFT: 3b783434f26b0e87994e6a77c5411927aad9ce3f
Tendermint Core: 66c2cb63416e66bff08e11f9088e21a0ed142790 | Pass | +| 2/3 Tendermint Core
1/3 CometBFT | 2023-02-08 | CometBFT: 3b783434f26b0e87994e6a77c5411927aad9ce3f
Tendermint Core: 66c2cb63416e66bff08e11f9088e21a0ed142790 | Pass | diff --git a/cometbft/v0.38/docs/qa/CometBFT-QA-37.mdx b/cometbft/v0.38/docs/qa/CometBFT-QA-37.mdx new file mode 100644 index 00000000..1181cf5d --- /dev/null +++ b/cometbft/v0.38/docs/qa/CometBFT-QA-37.mdx @@ -0,0 +1,157 @@ +--- +order: 1 +parent: + title: CometBFT QA Results v0.37.x + description: This is a report on the results obtained when running CometBFT v0.37.x on testnets + order: 5 +--- + +# CometBFT QA Results v0.37.x + +This iteration of the QA was run on CometBFT `v0.37.0-alpha3`, the first `v0.37.x` version from the CometBFT repository. + +The changes with respect to the baseline, `TM v0.37.x` as of Oct 12, 2022 (Commit: 1cf9d8e276afe8595cba960b51cd056514965fd1), include the rebranding of our fork of Tendermint Core to CometBFT and several improvements, described in the CometBFT [CHANGELOG](https://github.com/cometbft/cometbft/blob/v0.37.0-alpha.3/CHANGELOG.md). + +## Testbed + +As in other iterations of our QA process, we have used a 200-node network as testbed, plus nodes to introduce load and collect metrics. + +### Saturation point + +As in previous iterations, in our QA experiments, the system is subjected to a load slightly under a saturation point. +The method to identify the saturation point is explained [here](TMCore-QA-34.md#finding-the-saturation-point) and its application to the baseline is described [here](TMCore-QA-37.md#finding-the-saturation-point). +We use the same saturation point, that is, `c`, the number of connections created by the load runner process to the target node, is 2 and `r`, the rate or number of transactions issued per second, is 200. + +## Examining latencies + +The following figure plots six experiments carried out with the network. +Unique identifiers, UUID, for each execution are presented on top of each graph. + +![latencies](img37/200nodes_cmt037/all_experiments.png) + +We can see that the latencies follow comparable patterns across all experiments. +Therefore, in the following sections we will only present the results for one representative run, chosen randomly, with UUID starting with `75cb89a8`. + +![latencies](img37/200nodes_cmt037/e_75cb89a8-f876-4698-82f3-8aaab0b361af.png). + +For reference, the following figure shows the latencies of different configuration of the baseline. +`c=02 r=200` corresponds to the same configuration as in this experiment. + +![all-latencies](img37/200nodes_tm037/v037_200node_latencies.png) + +As can be seen, latencies are similar. + +## Prometheus Metrics on the Chosen Experiment + +This section further examines key metrics for this experiment extracted from Prometheus data regarding the chosen experiment. + +### Mempool Size + +The mempool size, a count of the number of transactions in the mempool, was shown to be stable and homogeneous at all full nodes. +It did not exhibit any unconstrained growth. +The plot below shows the evolution over time of the cumulative number of transactions inside all full nodes' mempools at a given time. + +![mempoool-cumulative](img37/200nodes_cmt037/mempool_size.png) + +The following picture shows the evolution of the average mempool size over all full nodes, which mostly oscilates between 1500 and 2000 outstanding transactions. + +![mempool-avg](img37/200nodes_cmt037/avg_mempool_size.png) + +The peaks observed coincide with the moments when some nodes reached round 1 of consensus (see below). + + +The behavior is similar to the observed in the baseline, presented next. + +![mempool-cumulative-baseline](img37/200nodes_tm037/mempool_size.png) + +![mempool-avg-baseline](img37/200nodes_tm037/avg_mempool_size.png) + + +### Peers + +The number of peers was stable at all nodes. +It was higher for the seed nodes (around 140) than for the rest (between 16 and 78). +The red dashed line denotes the average value. + +![peers](img37/200nodes_cmt037/peers.png) + +Just as in the baseline, shown next, the fact that non-seed nodes reach more than 50 peers is due to [\#9548]. + +![peers](img37/200nodes_tm037/peers.png) + + +### Consensus Rounds per Height + +Most heights took just one round, that is, round 0, but some nodes needed to advance to round 1 and eventually round 2. + +![rounds](img37/200nodes_cmt037/rounds.png) + +The following specific run of the baseline presented better results, only requiring up to round 1, but reaching higher rounds is not uncommon in the corresponding software version. + +![rounds](img37/200nodes_tm037/rounds.png) + +### Blocks Produced per Minute, Transactions Processed per Minute + +The following plot shows the rate in which blocks were created, from the point of view of each node. +That is, it shows when each node learned that a new block had been agreed upon. + +![heights](img37/200nodes_cmt037/block_rate.png) + +For most of the time when load was being applied to the system, most of the nodes stayed around 20 to 25 blocks/minute. + +The spike to more than 175 blocks/minute is due to a slow node catching up. + +The collective spike on the right of the graph marks the end of the load injection, when blocks become smaller (empty) and impose less strain on the network. +This behavior is reflected in the following graph, which shows the number of transactions processed per minute. + +![total-txs](img37/200nodes_cmt037/total_txs_rate.png) + +The baseline experienced a similar behavior, shown in the following two graphs. +The first depicts the block rate. + +![heights-baseline](img37/200nodes_tm037/block_rate_regular.png) + +The second plots the transaction rate. + +![total-txs-baseline](img37/200nodes_tm037/total_txs_rate_regular.png) + +### Memory Resident Set Size + +The Resident Set Size of all monitored processes is plotted below, with maximum memory usage of 2GB. + +![rss](img37/200nodes_cmt037/memory.png) + +A similar behavior was shown in the baseline, presented next. + +![rss](img37/200nodes_tm037/memory.png) + +The memory of all processes went down as the load as removed, showing no signs of unconstrained growth. + + +#### CPU utilization + +The best metric from Prometheus to gauge CPU utilization in a Unix machine is `load1`, +as it usually appears in the +[output of `top`](https://www.digitalocean.com/community/tutorials/load-average-in-linux). + +It is contained below 5 on most nodes, as seen in the following graph. + +![load1](img37/200nodes_cmt037/cpu.png) + +A similar behavior was seen in the baseline. + +![load1-baseline](img37/200nodes_tm037/cpu.png) + + +## Test Results + +The comparison against the baseline results show that both scenarios had similar numbers and are therefore equivalent. + +A conclusion of these tests is shown in the following table, along with the commit versions used in the experiments. + +| Scenario | Date | Version | Result | +|--|--|--|--| +|CometBFT | 2023-02-14 | v0.37.0-alpha3 (bef9a830e7ea7da30fa48f2cc236b1f465cc5833) | Pass + + +[\#9548]: https://github.com/tendermint/tendermint/issues/9548 diff --git a/cometbft/v0.38/docs/qa/CometBFT-QA-38.mdx b/cometbft/v0.38/docs/qa/CometBFT-QA-38.mdx new file mode 100644 index 00000000..b44b386c --- /dev/null +++ b/cometbft/v0.38/docs/qa/CometBFT-QA-38.mdx @@ -0,0 +1,556 @@ +--- +order: 1 +parent: + title: CometBFT QA Results v0.38.x + description: This is a report on the results obtained when running CometBFT v0.38.x on testnets + order: 5 +--- + +# CometBFT QA Results v0.38.x + +This iteration of the QA was run on CometBFT `v0.38.0-alpha.2`, the second +`v0.38.x` version from the CometBFT repository. + +The changes with respect to the baseline, `v0.37.0-alpha.3` from Feb 21, 2023, +include the introduction of the `FinalizeBlock` method to complete the full +range of ABCI++ functionality (ABCI 2.0), and other several improvements +described in the +[CHANGELOG](https://github.com/cometbft/cometbft/blob/v0.38.0-alpha.2/CHANGELOG.md). + +## Issues discovered + +* (critical, fixed) [\#539] and [\#546] - This bug causes the proposer to crash in + `PrepareProposal` because it does not have extensions while it should. + This happens mainly when the proposer was catching up. +* (critical, fixed) [\#562] - There were several bugs in the metrics-related + logic that were causing panics when the testnets were started. + +## 200 Node Testnet + +As in other iterations of our QA process, we have used a 200-node network as +testbed, plus nodes to introduce load and collect metrics. + +### Saturation point + +As in previous iterations of our QA experiments, we first find the transaction +load on which the system begins to show a degraded performance. Then we run the +experiments with the system subjected to a load slightly under the saturation +point. The method to identify the saturation point is explained +[here](CometBFT-QA-34.md#saturation-point) and its application to the baseline +is described [here](TMCore-QA-37.md#finding-the-saturation-point). + +The following table summarizes the results for the different experiments +(extracted from +[`v038_report_tabbed.txt`](img38/200nodes/v038_report_tabbed.txt)). The X axis +(`c`) is the number of connections created by the load runner process to the +target node. The Y axis (`r`) is the rate or number of transactions issued per +second. + +| | c=1 | c=2 | c=4 | +| ------ | --------: | --------: | ----: | +| r=200 | 17800 | **33259** | 33259 | +| r=400 | **35600** | 41565 | 41384 | +| r=800 | 36831 | 38686 | 40816 | +| r=1600 | 40600 | 45034 | 39830 | + +We can observe in the table that the system is saturated beyond the diagonal +defined by the entries `c=1,r=400` and `c=2,r=200`. Entries in the diagonal have +the same amount of transaction load, so we can consider them equivalent. For the +chosen diagonal, the expected number of processed transactions is `1 * 400 tx/s * 89 s = 35600`. +(Note that we use 89 out of 90 seconds of the experiment because the last transaction batch +coincides with the end of the experiment and is thus not sent.) The experiments in the diagonal +below expect double that number, that is, `1 * 800 tx/s * 89 s = 71200`, but the +system is not able to process such load, thus it is saturated. + +Therefore, for the rest of these experiments, we chose `c=1,r=400` as the +configuration. We could have chosen the equivalent `c=2,r=200`, which is the same +used in our baseline version, but for simplicity we decided to use the one with +only one connection. + +Also note that, compared to the previous QA tests, we have tried to find the +saturation point within a higher range of load values for the rate `r`. In +particular we run tests with `r` equal to or above `200`, while in the previous +tests `r` was `200` or lower. In particular, for our baseline version we didn't +run the experiment on the configuration `c=1,r=400`. + +For comparison, this is the table with the baseline version, where the +saturation point is beyond the diagonal defined by `r=200,c=2` and `r=100,c=4`. + +| | c=1 | c=2 | c=4 | +| ----- | ----: | --------: | --------: | +| r=25 | 2225 | 4450 | 8900 | +| r=50 | 4450 | 8900 | 17800 | +| r=100 | 8900 | 17800 | **35600** | +| r=200 | 17800 | **35600** | 38660 | + +### Latencies + +The following figure plots the latencies of the experiment carried out with the +configuration `c=1,r=400`. + +![latency-1-400](img38/200nodes/e_de676ecf-038e-443f-a26a-27915f29e312.png). + +For reference, the following figure shows the latencies of one of the +experiments for `c=2,r=200` in the baseline. + +![latency-2-200-37](img37/200nodes_cmt037/e_75cb89a8-f876-4698-82f3-8aaab0b361af.png) + +As can be seen, in most cases the latencies are very similar, and in some cases, +the baseline has slightly higher latencies than the version under test. Thus, +from this small experiment, we can say that the latencies measured on the two +versions are equivalent, or at least that the version under test is not worse +than the baseline. + +### Prometheus Metrics on the Chosen Experiment + +This section further examines key metrics for this experiment extracted from +Prometheus data regarding the chosen experiment with configuration `c=1,r=400`. + +#### Mempool Size + +The mempool size, a count of the number of transactions in the mempool, was +shown to be stable and homogeneous at all full nodes. It did not exhibit any +unconstrained growth. The plot below shows the evolution over time of the +cumulative number of transactions inside all full nodes' mempools at a given +time. + +![mempoool-cumulative](img38/200nodes/mempool_size.png) + +The following picture shows the evolution of the average mempool size over all +full nodes, which mostly oscilates between 1000 and 2500 outstanding +transactions. + +![mempool-avg](img38/200nodes/avg_mempool_size.png) + +The peaks observed coincide with the moments when some nodes reached round 1 of +consensus (see below). + +The behavior is similar to the observed in the baseline, presented next. + +![mempool-cumulative-baseline](img37/200nodes_cmt037/mempool_size.png) + +![mempool-avg-baseline](img37/200nodes_cmt037/avg_mempool_size.png) + + +#### Peers + +The number of peers was stable at all nodes. It was higher for the seed nodes +(around 140) than for the rest (between 20 and 70 for most nodes). The red +dashed line denotes the average value. + +![peers](img38/200nodes/peers.png) + +Just as in the baseline, shown next, the fact that non-seed nodes reach more +than 50 peers is due to [\#9548]. + +![peers](img37/200nodes_cmt037/peers.png) + + +#### Consensus Rounds per Height + +Most heights took just one round, that is, round 0, but some nodes needed to +advance to round 1. + +![rounds](img38/200nodes/rounds.png) + +The following specific run of the baseline required some nodes to reach round 1. + +![rounds](img37/200nodes_cmt037/rounds.png) + + +#### Blocks Produced per Minute, Transactions Processed per Minute + +The following plot shows the rate in which blocks were created, from the point +of view of each node. That is, it shows when each node learned that a new block +had been agreed upon. + +![heights](img38/200nodes/block_rate.png) + +For most of the time when load was being applied to the system, most of the +nodes stayed around 20 blocks/minute. + +The spike to more than 100 blocks/minute is due to a slow node catching up. + +The baseline experienced a similar behavior. + +![heights-baseline](img37/200nodes_cmt037/block_rate.png) + +The collective spike on the right of the graph marks the end of the load +injection, when blocks become smaller (empty) and impose less strain on the +network. This behavior is reflected in the following graph, which shows the +number of transactions processed per minute. + +![total-txs](img38/200nodes/total_txs_rate.png) + +The following is the transaction processing rate of the baseline, which is +similar to above. + +![total-txs-baseline](img37/200nodes_cmt037/total_txs_rate.png) + + +#### Memory Resident Set Size + +The following graph shows the Resident Set Size of all monitored processes, with +maximum memory usage of 1.6GB, slightly lower than the baseline shown after. + +![rss](img38/200nodes/memory.png) + +A similar behavior was shown in the baseline, with even a slightly higher memory +usage. + +![rss](img37/200nodes_cmt037/memory.png) + +The memory of all processes went down as the load is removed, showing no signs +of unconstrained growth. + + +#### CPU utilization + +##### Comparison to baseline + +The best metric from Prometheus to gauge CPU utilization in a Unix machine is +`load1`, as it usually appears in the [output of +`top`](https://www.digitalocean.com/community/tutorials/load-average-in-linux). + +The load is contained below 5 on most nodes, as seen in the following graph. + +![load1](img38/200nodes/cpu.png) + +The baseline had a similar behavior. + +![load1-baseline](img37/200nodes_cmt037/cpu.png) + +##### Impact of vote extension signature verification + +It is important to notice that the baseline (`v0.37.x`) does not implement vote extensions, +whereas the version under test (`v0.38.0-alpha.2`) _does_ implement them, and they are +configured to be activated since height 1. +The e2e application used in these tests verifies all received vote extension signatures (up to 175) +twice per height: upon `PrepareProposal` (for sanity) and upon `ProcessProposal` (to demonstrate how +real applications can do it). + +The fact that there is no noticeable difference in the CPU utilization plots of +the baseline and `v0.38.0-alpha.2` means that re-verifying up 175 vote extension signatures twice +(besides the initial verification done by CometBFT when receiving them from the network) +has no performance impact in the current version of the system: the bottlenecks are elsewhere. +Thus, we should focus on optimizing other parts of the system: the ones that cause the current +bottlenecks (mempool gossip duplication, leaner proposal structure, optimized consensus gossip). + +### Test Results + +The comparison against the baseline results show that both scenarios had similar +numbers and are therefore equivalent. + +A conclusion of these tests is shown in the following table, along with the +commit versions used in the experiments. + +| Scenario | Date | Version | Result | +| -------- | ---------- | ---------------------------------------------------------- | ------ | +| 200-node | 2023-05-21 | v0.38.0-alpha.2 (1f524d12996204f8fd9d41aa5aca215f80f06f5e) | Pass | + + +## Rotating Node Testnet + +We use `c=1,r=400` as load, which can be considered a safe workload, as it was close to (but below) +the saturation point in the 200 node testnet. This testnet has less nodes (10 validators and 25 full nodes). + +Importantly, the baseline considered in this section is `v0.37.0-alpha.2` (Tendermint Core), +which is **different** from the one used in the [previous section](method.md#200-node-testnet). +The reason is that this testnet was not re-tested for `v0.37.0-alpha.3` (CometBFT), +since it was not deemed necessary. + +Unlike in the baseline tests, the version of CometBFT used for these tests is _not_ affected by [\#9539], +which was fixed right after having run rotating testnet for `v0.37`. +As a result, the load introduced in this iteration of the test is higher as transactions do not get rejected. + +### Latencies + +The plot of all latencies can be seen here. + +![rotating-all-latencies](img38/rotating/rotating_latencies.png) + +Which is similar to the baseline. + +![rotating-all-latencies](img37/200nodes_tm037/v037_rotating_latencies.png) + +The average increase of about 1 second with respect to the baseline is due to the higher +transaction load produced (remember the baseline was affected by [\#9539], whereby most transactions +produced were rejected by `CheckTx`). + +### Prometheus Metrics + +The set of metrics shown here roughly match those shown on the baseline (`v0.37`) for the same experiment. +We also show the baseline results for comparison. + +#### Blocks and Transactions per minute + +This following plot shows the blocks produced per minute. + +![rotating-heights](img38/rotating/rotating_block_rate.png) + +This is similar to the baseline, shown below. + +![rotating-heights-bl](img37/rotating/rotating_block_rate.png) + +The following plot shows only the heights reported by ephemeral nodes, both when they were blocksyncing +and when they were running consensus. +The second plot is the baseline plot for comparison. The baseline lacks the heights when the nodes were +blocksyncing as that metric was implemented afterwards. + +![rotating-heights-ephe](img38/rotating/rotating_eph_heights.png) + +![rotating-heights-ephe-bl](img37/rotating/rotating_eph_heights.png) + +We seen that heights follow a similar pattern in both plots: they grow in length as the experiment advances. + +The following plot shows the transactions processed per minute. + +![rotating-total-txs](img38/rotating/rotating_txs_rate.png) + +For comparison, this is the baseline plot. + +![rotating-total-txs-bl](img37/rotating/rotating_txs_rate.png) + +We can see the rate is much lower in the baseline plot. +The reason is that the baseline was affected by [\#9539], whereby `CheckTx` rejected most transactions +produced by the load runner. + +#### Peers + +The plot below shows the evolution of the number of peers throughout the experiment. + +![rotating-peers](img38/rotating/rotating_peers.png) + +This is the baseline plot, for comparison. + +![rotating-peers-bl](img37/rotating/rotating_peers.png) + +The plotted values and their evolution are comparable in both plots. + +For further details on these plots, see the [this section](./TMCore-QA-34.md#peers-1). + +#### Memory Resident Set Size + +The average Resident Set Size (RSS) over all processes is notably bigger on `v0.38.0-alpha.2` than on the baseline. +The reason for this is, again, the fact that `CheckTx` was rejecting most transactions submitted on the baseline +and therefore the overall transaction load was lower on the baseline. +This is consistent with the difference seen in the transaction rate plots +in the [previous section](#blocks-and-transactions-per-minute). + +![rotating-rss-avg](img38/rotating/rotating_avg_memory.png) + +![rotating-rss-avg-bl](img37/rotating/rotating_avg_memory.png) + +#### CPU utilization + +The plots show metric `load1` for all nodes for `v0.38.0-alpha.2` and for the baseline. + +![rotating-load1](img38/rotating/rotating_cpu.png) + +![rotating-load1-bl](img37/rotating/rotating_cpu.png) + +In both cases, it is contained under 5 most of the time, which is considered normal load. +The load seems to be more important on `v0.38.0-alpha.2` on average because of the bigger +number of transactions processed per minute as compared to the baseline. + +### Test Result + +| Scenario | Date | Version | Result | +| -------- | ---------- | ---------------------------------------------------------- | ------ | +| Rotating | 2023-05-23 | v0.38.0-alpha.2 (e9abb116e29beb830cf111b824c8e2174d538838) | Pass | + + + +## Vote Extensions Testbed + +In this testnet we evaluate the effect of varying the sizes of vote extensions added to pre-commit votes on the performance of CometBFT. +The test uses the Key/Value store in our [[end-to-end]] test framework, which has the following simplified flow: + +1. When validators send their pre-commit votes to a block of height $i$, they first extend the vote as they see fit in `ExtendVote`. +2. When a proposer for height $i+1$ creates a block to propose, in `PrepareProposal`, it prepends the transactions with a special transaction, which modifies a reserved key. The transaction value is derived from the extensions from height $i$; in this example, the value is derived from the vote extensions and includes the set itself, hexa encoded as string. +3. When a validator sends their pre-vote for the block proposed in $i+1$, they first double check in `ProcessProposal` that the special transaction in the block was properly built by the proposer. +4. When validators send their pre-commit for the block proposed in $i+1$, they first extend the vote, and the steps repeat for heights $i+2$ and so on. + +For this test, extensions are random sequences of bytes with a predefined `vote_extension_size`. +Hence, two effects are seen on the network. +First, pre-commit vote message sizes will increase by the specified `vote_extension_size` and, second, block messages will increase by twice `vote_extension_size`, given then hexa encoding of extensions, times the number of extensions received, i.e. at least 2/3 of 175. + +All tests were performed on commit d5baba237ab3a04c1fd4a7b10927ba2e6a2aab27, which corresponds to v0.38.0-alpha.2 plus commits to add the ability to vary the vote extension sizes to the test application. +Although the same commit is used for the baseline, in this configuration the behavior observed is the same as in the "vanilla" v0.38.0-alpha.2 test application, that is, vote extensions are 8-byte integers, compressed as variable size integers instead of a random sequence of size `vote_extension_size`. + +The following table summarizes the test cases. + +| Name | Extension Size (bytes) | Date | +| -------- | ---------------------- | ---------- | +| baseline | 8 (varint) | 2023-05-26 | +| 2k | 2048 | 2023-05-29 | +| 4k | 4094 | 2023-05-29 | +| 8k | 8192 | 2023-05-26 | +| 16k | 16384 | 2023-05-26 | +| 32k | 32768 | 2023-05-26 | + + +### Latency + +The following figures show the latencies observed on each of the 5 runs of each experiment; +the redline shows the average of each run. +It can be easily seen from these graphs that the larger the vote extension size, the more latency varies and the more common higher latencies become. +Even in the case of extensions of size 2k, the mean latency goes from below 5s to nearly 10s. + +**Baseline** + +![](img38/voteExtensions/all_experiments_baseline.png) + +**2k** + +![](img38/voteExtensions/all_experiments_2k.png) + +**4k** + +![](img38/voteExtensions/all_experiments_4k.png) + +**8k** + +![](img38/voteExtensions/all_experiments_8k.png) + +**16k** + +![](img38/voteExtensions/all_experiments_16k.png) + +**32k** + +![](img38/voteExtensions/all_experiments_32k.png) + +The following graphs combine all the runs of the same experiment. +They show that latency variation greatly increases with the increase of vote extensions. +In particular, for the 16k and 32k cases, the system goes through large gaps without transaction delivery. +As discussed later, this is the result of heights taking multiple rounds to finish and new transactions being held until the next block is agreed upon. + +| | | +| ---------------------------------------------------------- | ------------------------------------------------ | +| baseline ![](img38/voteExtensions/all_c1r400_baseline.png) | 2k ![](img38/voteExtensions/all_c1r400_2k.png) | +| 4k ![](img38/voteExtensions/all_c1r400_4k.png) | 8k ![](img38/voteExtensions/all_c1r400_8k.png) | +| 16k ![](img38/voteExtensions/all_c1r400_16k.png) | 32k ![](img38/voteExtensions/all_c1r400_32k.png) | + + +### Blocks and Transactions per minute + +The following plots show the blocks produced per minute and transactions processed per minute. +We have divided the presentation in an overview section, which shows the metrics for the whole experiment (five runs) and a detailed sample, which shows the metrics for the first of the five runs. +We repeat the approach for the other metrics as well. +The dashed red line shows the moving average over a 20s window. + +#### Overview + +It is clear from the overview plots that as the vote extension sizes increase, the rate of block creation decreases. +Although the rate of transaction processing also decreases, it does not seem to decrease as fast. + +| Experiment | Block creation rate | Transaction rate | +| ------------ | ----------------------------------------------------------- | ------------------------------------------------------------- | +| **baseline** | ![block rate](img38/voteExtensions/baseline_block_rate.png) | ![txs rate](img38/voteExtensions/baseline_total_txs_rate.png) | +| **2k** | ![block rate](img38/voteExtensions/02k_block_rate.png) | ![txs rate](img38/voteExtensions/02k_total_txs_rate.png) | +| **4k** | ![block rate](img38/voteExtensions/04k_block_rate.png) | ![txs rate](img38/voteExtensions/04k_total_txs_rate.png) | +| **8k** | ![block rate](img38/voteExtensions/8k_block_rate.png) | ![txs rate](img38/voteExtensions/08k_total_txs_rate.png) | +| **16k** | ![block rate](img38/voteExtensions/16k_block_rate.png) | ![txs rate](img38/voteExtensions/16k_total_txs_rate.png) | +| **32k** | ![block rate](img38/voteExtensions/32k_block_rate.png) | ![txs rate](img38/voteExtensions/32k_total_txs_rate.png) | + +#### First run + +| Experiment | Block creation rate | Transaction rate | +| ------------ | ------------------------------------------------------------- | --------------------------------------------------------------- | +| **baseline** | ![block rate](img38/voteExtensions/baseline_1_block_rate.png) | ![txs rate](img38/voteExtensions/baseline_1_total_txs_rate.png) | +| **2k** | ![block rate](img38/voteExtensions/02k_1_block_rate.png) | ![txs rate](img38/voteExtensions/02k_1_total_txs_rate.png) | +| **4k** | ![block rate](img38/voteExtensions/04k_1_block_rate.png) | ![txs rate](img38/voteExtensions/04k_1_total_txs_rate.png) | +| **8k** | ![block rate](img38/voteExtensions/08k_1_block_rate.png) | ![txs rate](img38/voteExtensions/08k_1_total_txs_rate.png) | +| **16k** | ![block rate](img38/voteExtensions/16k_1_block_rate.png) | ![txs rate](img38/voteExtensions/16k_1_total_txs_rate.png) | +| **32k** | ![block rate](img38/voteExtensions/32k_1_block_rate.png) | ![txs rate](img38/voteExtensions/32k_1_total_txs_rate.png) | + + +### Number of rounds + +The effect of vote extensions are also felt on the number of rounds needed to reach consensus. +The following graphs show the number of the highest round required to reach consensus during the whole experiment. + +In the baseline and low vote extension lengths, most blocks were agreed upon during round 0. +As the load increases, more and more rounds were required. +In the 32k case se see round 5 being reached frequently. + +| Experiment | Number of Rounds per block | +| ------------ | ------------------------------------------------------------- | +| **baseline** | ![number of rounds](img38/voteExtensions/baseline_rounds.png) | +| **2k** | ![number of rounds](img38/voteExtensions/02k_rounds.png) | +| **4k** | ![number of rounds](img38/voteExtensions/04k_rounds.png) | +| **8k** | ![number of rounds](img38/voteExtensions/08k_rounds.png) | +| **16k** | ![number of rounds](img38/voteExtensions/16k_rounds.png) | +| **32k** | ![number of rounds](img38/voteExtensions/32k_rounds.png) | + + +We conjecture that the reason is that the timeouts used are inadequate for the extra traffic in the network. + +### CPU + +The CPU usage reached the same peaks on all tests, but the following graphs show that with larger Vote Extensions, nodes take longer to reduce the CPU usage. +This could mean that a backlog of processing is forming during the execution of the tests with larger extensions. + + +| Experiment | CPU | +| ------------ | ----------------------------------------------------- | +| **baseline** | ![cpu-avg](img38/voteExtensions/baseline_avg_cpu.png) | +| **2k** | ![cpu-avg](img38/voteExtensions/02k_avg_cpu.png) | +| **4k** | ![cpu-avg](img38/voteExtensions/04k_avg_cpu.png) | +| **8k** | ![cpu-avg](img38/voteExtensions/08k_avg_cpu.png) | +| **16k** | ![cpu-avg](img38/voteExtensions/16k_avg_cpu.png) | +| **32k** | ![cpu-avg](img38/voteExtensions/32k_avg_cpu.png) | + +### Resident Memory + +The same conclusion reached for CPU usage may be drawn for the memory. +That is, that a backlog of work is formed during the tests and catching up (freeing of memory) happens after the test is done. + +A more worrying trend is that the bottom of the memory usage seems to increase in between runs. +We have investigated this in longer runs and confirmed that there is no such a trend. + + + +| Experiment | Resident Set Size | +| ------------ | -------------------------------------------------------- | +| **baseline** | ![rss-avg](img38/voteExtensions/baseline_avg_memory.png) | +| **2k** | ![rss-avg](img38/voteExtensions/02k_avg_memory.png) | +| **4k** | ![rss-avg](img38/voteExtensions/04k_avg_memory.png) | +| **8k** | ![rss-avg](img38/voteExtensions/08k_avg_memory.png) | +| **16k** | ![rss-avg](img38/voteExtensions/16k_avg_memory.png) | +| **32k** | ![rss-avg](img38/voteExtensions/32k_avg_memory.png) | + +### Mempool size + +This metric shows how many transactions are outstanding in the nodes' mempools. +Observe that in all runs, the average number of transactions in the mempool quickly drops to near zero between runs. + + +| Experiment | Resident Set Size | +| ------------ | ------------------------------------------------------------------ | +| **baseline** | ![mempool-avg](img38/voteExtensions/baseline_avg_mempool_size.png) | +| **2k** | ![mempool-avg](img38/voteExtensions/02k_avg_mempool_size.png) | +| **4k** | ![mempool-avg](img38/voteExtensions/04k_avg_mempool_size.png) | +| **8k** | ![mempool-avg](img38/voteExtensions/08k_avg_mempool_size.png) | +| **16k** | ![mempool-avg](img38/voteExtensions/16k_avg_mempool_size.png) | +| **32k** | ![mempool-avg](img38/voteExtensions/32k_avg_mempool_size.png) | + + + + + +### Results + +| Scenario | Date | Version | Result | +| -------- | ---------- | ------------------------------------------------------------------------------------- | ------ | +| VESize | 2023-05-23 | v0.38.0-alpha.2 + varying vote extensions (9fc711b6514f99b2dc0864fc703cb81214f01783) | N/A | + + + +[\#9539]: https://github.com/tendermint/tendermint/issues/9539 +[\#9548]: https://github.com/tendermint/tendermint/issues/9548 +[\#539]: https://github.com/cometbft/cometbft/issues/539 +[\#546]: https://github.com/cometbft/cometbft/issues/546 +[\#562]: https://github.com/cometbft/cometbft/issues/562 +[end-to-end]: https://github.com/cometbft/cometbft/tree/main/test/e2e diff --git a/cometbft/v0.38/docs/qa/CometBFT-QA.mdx b/cometbft/v0.38/docs/qa/CometBFT-QA.mdx new file mode 100644 index 00000000..a9b678f8 --- /dev/null +++ b/cometbft/v0.38/docs/qa/CometBFT-QA.mdx @@ -0,0 +1,26 @@ +--- +order: 1 +parent: + title: CometBFT Quality Assurance + description: This is a report on the process followed and results obtained when running v0.34.x on testnets + order: 6 +--- + +# CometBFT Quality Assurance + +This directory keeps track of the process followed by the CometBFT team +for Quality Assurance before cutting a release. +This directory is to live in multiple branches. On each release branch, +the contents of this directory reflect the status of the process +at the time the Quality Assurance process was applied for that release. + +File [method](./method.md) keeps track of the process followed to obtain the results +used to decide if a release is passing the Quality Assurance process. +The results obtained in each release are stored in their own directory. +The following releases have undergone the Quality Assurance process, and the corresponding reports include detailed information on tests and comparison with the baseline. + +* [TM v0.34.x](TMCore-QA-34.md) - Tested prior to releasing Tendermint Core v0.34.22. +* [v0.34.x](CometBFT-QA-34.md) - Tested prior to releasing v0.34.27, using TM v0.34.x results as baseline. +* [TM v0.37.x](TMCore-QA-37.md) - Tested prior to releasing TM v0.37.x, using TM v0.34.x results as baseline. +* [v0.37.x](CometBFT-QA-37.md) - Tested on CometBFT v0.37.0-alpha3, using TM v0.37.x results as baseline. +* [v0.38.x](CometBFT-QA-38.md) - Tested on v0.38.0-alpha.2, using v0.37.x results as baseline. diff --git a/cometbft/v0.38/docs/qa/Method.mdx b/cometbft/v0.38/docs/qa/Method.mdx new file mode 100644 index 00000000..a1663f65 --- /dev/null +++ b/cometbft/v0.38/docs/qa/Method.mdx @@ -0,0 +1,262 @@ +--- +order: 1 +parent: + title: Method + order: 1 +--- + +# Method + +This document provides a detailed description of the QA process. +It is intended to be used by engineers reproducing the experimental setup for future tests of CometBFT. + +The (first iteration of the) QA process as described [in the RELEASES.md document][releases] +was applied to version v0.34.x in order to have a set of results acting as benchmarking baseline. +This baseline is then compared with results obtained in later versions. + +Out of the testnet-based test cases described in [the releases document][releases] we focused on two of them: +_200 Node Test_, and _Rotating Nodes Test_. + +[releases]: https://github.com/cometbft/cometbft/blob/v0.38.x/RELEASES.md#large-scale-testnets + +## Software Dependencies + +### Infrastructure Requirements to Run the Tests + +* An account at Digital Ocean (DO), with a high droplet limit (>202) +* The machine to orchestrate the tests should have the following installed: + * A clone of the [testnet repository][testnet-repo] + * This repository contains all the scripts mentioned in the remainder of this section + * [Digital Ocean CLI][doctl] + * [Terraform CLI][Terraform] + * [Ansible CLI][Ansible] + +[testnet-repo]: https://github.com/cometbft/qa-infra +[Ansible]: https://docs.ansible.com/ansible/latest/index.html +[Terraform]: https://www.terraform.io/docs +[doctl]: https://docs.digitalocean.com/reference/doctl/how-to/install/ + +### Requirements for Result Extraction + +* [Prometheus DB][prometheus] to collect metrics from nodes +* Prometheus DB to process queries (may be different node from the previous) +* blockstore DB of one of the full nodes in the testnet + + +[prometheus]: https://prometheus.io/ + +## 200 Node Testnet + +### Running the test + +This section explains how the tests were carried out for reproducibility purposes. + +1. [If you haven't done it before] + Follow steps 1-4 of the `README.md` at the top of the testnet repository to configure Terraform, and `doctl`. +2. Copy file `testnets/testnet200.toml` onto `testnet.toml` (do NOT commit this change) +3. Set the variable `VERSION_TAG` in the `Makefile` to the git hash that is to be tested. + * If you are running the base test, which implies an homogeneous network (all nodes are running the same version), + then make sure makefile variable `VERSION2_WEIGHT` is set to 0 + * If you are running a mixed network, set the variable `VERSION2_TAG` to the other version you want deployed + in the network. + Then adjust the weight variables `VERSION_WEIGHT` and `VERSION2_WEIGHT` to configure the + desired proportion of nodes running each of the two configured versions. +4. Follow steps 5-10 of the `README.md` to configure and start the 200 node testnet + * WARNING: Do NOT forget to run `make terraform-destroy` as soon as you are done with the tests (see step 9) +5. As a sanity check, connect to the Prometheus node's web interface (port 9090) + and check the graph for the `cometbft_consensus_height` metric. All nodes + should be increasing their heights. + + * You can find the Prometheus node's IP address in `ansible/hosts` under section `[prometheus]`. + * The following URL will display the metrics `cometbft_consensus_height` and `cometbft_mempool_size`: + + ``` + http://:9090/classic/graph?g0.range_input=1h&g0.expr=cometbft_consensus_height&g0.tab=0&g1.range_input=1h&g1.expr=cometbft_mempool_size&g1.tab=0 + ``` + +6. You now need to start the load runner that will produce transaction load. + * If you don't know the saturation load of the version you are testing, you need to discover it. + * Run `make loadrunners-init`. This will copy the loader scripts to the + `testnet-load-runner` node and install the load tool. + * Find the IP address of the `testnet-load-runner` node in + `ansible/hosts` under section `[loadrunners]`. + * `ssh` into `testnet-load-runner`. + * Edit the script `/root/200-node-loadscript.sh` in the load runner + node to provide the IP address of a full node (for example, + `validator000`). This node will receive all transactions from the + load runner node. + * Run `/root/200-node-loadscript.sh` from the load runner node. + * This script will take about 40 mins to run, so it is suggested to + first run `tmux` in case the ssh session breaks. + * It is running 90-seconds-long experiments in a loop with different + loads. + * If you already know the saturation load, you can simply run the test (several times) for 90 seconds with a load somewhat + below saturation: + * set makefile variables `LOAD_CONNECTIONS`, `LOAD_TX_RATE`, to values that will produce the desired transaction load. + * set `LOAD_TOTAL_TIME` to 90 (seconds). + * run "make runload" and wait for it to complete. You may want to run this several times so the data from different runs can be compared. +7. Run `make retrieve-data` to gather all relevant data from the testnet into the orchestrating machine + * Alternatively, you may want to run `make retrieve-prometheus-data` and `make retrieve-blockstore` separately. + The end result will be the same. + * `make retrieve-blockstore` accepts the following values in makefile variable `RETRIEVE_TARGET_HOST` + * `any`: (which is the default) picks up a full node and retrieves the blockstore from that node only. + * `all`: retrieves the blockstore from all full nodes; this is extremely slow, and consumes plenty of bandwidth, + so use it with care. + * the name of a particular full node (e.g., `validator01`): retrieves the blockstore from that node only. +8. Verify that the data was collected without errors + * at least one blockstore DB for a CometBFT validator + * the Prometheus database from the Prometheus node + * for extra care, you can run `zip -T` on the `prometheus.zip` file and (one of) the `blockstore.db.zip` file(s) +9. **Run `make terraform-destroy`** + * Don't forget to type `yes`! Otherwise you're in trouble. + +### Result Extraction + +The method for extracting the results described here is highly manual (and exploratory) at this stage. +The CometBFT team should improve it at every iteration to increase the amount of automation. + +#### Steps + +1. Unzip the blockstore into a directory +2. To identify saturation points + 1. Extract the latency report for all the experiments. + * Run these commands from the directory containing the `blockstore.db` folder. + * It is advisable to adjust the hash in the `go run` command to the latest possible. + * ```bash + mkdir results + go run github.com/cometbft/cometbft/test/loadtime/cmd/report@3003ef7 --database-type goleveldb --data-dir ./ > results/report.txt + ``` + 2. File `report.txt` contains an unordered list of experiments with varying concurrent connections and transaction rate. + You will need to separate data per experiment. + + * Create files `report01.txt`, `report02.txt`, `report04.txt` and, for each experiment in file `report.txt`, + copy its related lines to the filename that matches the number of connections, for example + + ```bash + for cnum in 1 2 4; do echo "$cnum"; grep "Connections: $cnum" results/report.txt -B 2 -A 10 > results/report$cnum.txt; done + ``` + + * Sort the experiments in `report01.txt` in ascending tx rate order. Likewise for `report02.txt` and `report04.txt`. + * Otherwise just keep `report.txt`, and skip to the next step. + 4. Generate file `report_tabbed.txt` by showing the contents `report01.txt`, `report02.txt`, `report04.txt` side by side + * This effectively creates a table where rows are a particular tx rate and columns are a particular number of websocket connections. + * Combine the column files into a single table file: + * Replace tabs by spaces in all column files. For example, + `sed -i.bak 's/\t/ /g' results/report1.txt`. + * Merge the new column files into one: + `paste results/report1.txt results/report2.txt results/report4.txt | column -s $'\t' -t > report_tabbed.txt` + +3. To generate a latency vs throughput plot, extract the data as a CSV + * ```bash + go run github.com/cometbft/cometbft/test/loadtime/cmd/report@3003ef7 --database-type goleveldb --data-dir ./ --csv results/raw.csv + ``` + * Follow the instructions for the [`latency_throughput.py`] script. + This plot is useful to visualize the saturation point. + * Alternatively, follow the instructions for the [`latency_plotter.py`] script. + This script generates a series of plots per experiment and configuration that may + help with visualizing Latency vs Throughput variation. + +[`latency_throughput.py`]: https://github.com/cometbft/cometbft/tree/v0.38.x/scripts/qa/reporting#latency-vs-throughput-plotting +[`latency_plotter.py`]: https://github.com/cometbft/cometbft/tree/v0.38.x/scripts/qa/reporting#latency-vs-throughput-plotting-version-2 + +#### Extracting Prometheus Metrics + +1. Stop the prometheus server if it is running as a service (e.g. a `systemd` unit). +2. Unzip the prometheus database retrieved from the testnet, and move it to replace the + local prometheus database. +3. Start the prometheus server and make sure no error logs appear at start up. +4. Identify the time window you want to plot in your graphs. +5. Execute the [`prometheus_plotter.py`] script for the time window. + +[`prometheus_plotter.py`]: https://github.com/cometbft/cometbft/tree/v0.38.x/scripts/qa/reporting#prometheus-metrics + +## Rotating Node Testnet + +### Running the test + +This section explains how the tests were carried out for reproducibility purposes. + +1. [If you haven't done it before] + Follow steps 1-4 of the `README.md` at the top of the testnet repository to configure Terraform, and `doctl`. +2. Copy file `testnet_rotating.toml` onto `testnet.toml` (do NOT commit this change) +3. Set variable `VERSION_TAG` to the git hash that is to be tested. +4. Run `make terraform-apply EPHEMERAL_SIZE=25` + * WARNING: Do NOT forget to run `make terraform-destroy` as soon as you are done with the tests +5. Follow steps 6-10 of the `README.md` to configure and start the "stable" part of the rotating node testnet +6. As a sanity check, connect to the Prometheus node's web interface and check the graph for the `tendermint_consensus_height` metric. + All nodes should be increasing their heights. +7. On a different shell, + * run `make runload LOAD_CONNECTIONS=X LOAD_TX_RATE=Y LOAD_TOTAL_TIME=Z` + * `X` and `Y` should reflect a load below the saturation point (see, e.g., + [this paragraph](./TMCore-QA-34.md#finding-the-saturation-point) for further info) + * `Z` (in seconds) should be big enough to keep running throughout the test, until we manually stop it in step 9. + In principle, a good value for `Z` is `7200` (2 hours) +8. Run `make rotate` to start the script that creates the ephemeral nodes, and kills them when they are caught up. + * WARNING: If you run this command from your laptop, the laptop needs to be up and connected for the full length + of the experiment. + * [This](http://:9090/classic/graph?g0.range_input=100m&g0.expr=cometbft_consensus_height%7Bjob%3D~%22ephemeral.*%22%7D%20or%20cometbft_blocksync_latest_block_height%7Bjob%3D~%22ephemeral.*%22%7D&g0.tab=0&g1.range_input=100m&g1.expr=cometbft_mempool_size%7Bjob!~%22ephemeral.*%22%7D&g1.tab=0&g2.range_input=100m&g2.expr=cometbft_consensus_num_txs%7Bjob!~%22ephemeral.*%22%7D&g2.tab=0) + is an example Prometheus URL you can use to monitor the test case's progress +9. When the height of the chain reaches 3000, stop the `make runload` script. +10. When the rotate script has made two iterations (i.e., all ephemeral nodes have caught up twice) + after height 3000 was reached, stop `make rotate` +11. Run `make stop-network` +12. Run `make retrieve-data` to gather all relevant data from the testnet into the orchestrating machine +13. Verify that the data was collected without errors + * at least one blockstore DB for a CometBFT validator + * the Prometheus database from the Prometheus node + * for extra care, you can run `zip -T` on the `prometheus.zip` file and (one of) the `blockstore.db.zip` file(s) +14. **Run `make terraform-destroy`** + +Steps 8 to 10 are highly manual at the moment and will be improved in next iterations. + +### Result Extraction + +In order to obtain a latency plot, follow the instructions above for the 200 node experiment, +but the `results.txt` file contains only one experiment. + +As for prometheus, the same method as for the 200 node experiment can be applied. + +## Vote Extensions Testnet + +### Running the test + +This section explains how the tests were carried out for reproducibility purposes. + +1. [If you haven't done it before] + Follow steps 1-4 of the `README.md` at the top of the testnet repository to configure Terraform, and `doctl`. +2. Copy file `varyVESize.toml` onto `testnet.toml` (do NOT commit this change). +3. Set variable `VERSION_TAG` in the `Makefile` to the git hash that is to be tested. +4. Follow steps 5-10 of the `README.md` to configure and start the testnet + * WARNING: Do NOT forget to run `make terraform-destroy` as soon as you are done with the tests +5. Configure the load runner to produce the desired transaction load. + * set makefile variables `ROTATE_CONNECTIONS`, `ROTATE_TX_RATE`, to values that will produce the desired transaction load. + * set `ROTATE_TOTAL_TIME` to 150 (seconds). + * set `ITERATIONS` to the number of iterations that each configuration should run for. +6. Execute steps 5-10 of the `README.md` file at the testnet repository. + +7. Repeat the following steps for each desired `vote_extension_size` + 1. Update the configuration (you can skip this step if you didn't change the `vote_extension_size`) + * Update the `vote_extensions_size` in the `testnet.toml` to the desired value. + * `make configgen` + * `ANSIBLE_SSH_RETRIES=10 ansible-playbook ./ansible/re-init-testapp.yaml -u root -i ./ansible/hosts --limit=validators -e "testnet_dir=testnet" -f 20` + * `make restart` + 2. Run the test + * `make runload` + This will repeat the tests `ITERATIONS` times every time it is invoked. + 3. Collect your data + * `make retrieve-data` + Gathers all relevant data from the testnet into the orchestrating machine, inside folder `experiments`. + Two subfolders are created, one blockstore DB for a CometBFT validator and one for the Prometheus DB data. + * Verify that the data was collected without errors with `zip -T` on the `prometheus.zip` file and (one of) the `blockstore.db.zip` file(s). +8. Clean up your setup. + * `make terraform-destroy`; don't forget that you need to type **yes** for it to complete. + + +### Result Extraction + +In order to obtain a latency plot, follow the instructions above for the 200 node experiment, but: + +* The `results.txt` file contains only one experiment +* Therefore, no need for any `for` loops + +As for Prometheus, the same method as for the 200 node experiment can be applied. diff --git a/cometbft/v0.38/docs/qa/TMCore-QA-34.mdx b/cometbft/v0.38/docs/qa/TMCore-QA-34.mdx new file mode 100644 index 00000000..e5764611 --- /dev/null +++ b/cometbft/v0.38/docs/qa/TMCore-QA-34.mdx @@ -0,0 +1,277 @@ +--- +order: 1 +parent: + title: Tendermint Core QA Results v0.34.x + description: This is a report on the results obtained when running v0.34.x on testnets + order: 2 +--- + +# Tendermint Core QA Results v0.34.x + +## 200 Node Testnet + +### Finding the Saturation Point + +The first goal when examining the results of the tests is identifying the saturation point. +The saturation point is a setup with a transaction load big enough to prevent the testnet +from being stable: the load runner tries to produce slightly more transactions than can +be processed by the testnet. + +The following table summarizes the results for v0.34.x, for the different experiments +(extracted from file [`v034_report_tabbed.txt`](img34/v034_report_tabbed.txt)). + +The X axis of this table is `c`, the number of connections created by the load runner process to the target node. +The Y axis of this table is `r`, the rate or number of transactions issued per second. + +| | c=1 | c=2 | c=4 | +| :--- | ----: | ----: | ----: | +| r=25 | 2225 | 4450 | 8900 | +| r=50 | 4450 | 8900 | 17800 | +| r=100 | 8900 | 17800 | 35600 | +| r=200 | 17800 | 35600 | 38660 | + +The table shows the number of 1024-byte-long transactions that were produced by the load runner, +and processed by Tendermint Core, during the 90 seconds of the experiment's duration. +Each cell in the table refers to an experiment with a particular number of websocket connections (`c`) +to a chosen validator, and the number of transactions per second that the load runner +tries to produce (`r`). Note that the overall load that the tool attempts to generate is $c \cdot r$. + +We can see that the saturation point is beyond the diagonal that spans cells + +* `r=200,c=2` +* `r=100,c=4` + +given that the total number of transactions should be close to the product rate X the number of connections x experiment time. + +All experiments below the saturation diagonal (`r=200,c=4`) have in common that the total +number of transactions processed is noticeably less than the product $c \cdot r \cdot 89$ (89 seconds, since the last batch never gets sent), +which is the expected number of transactions when the system is able to deal well with the +load. +With (`r=200,c=4`), we obtained 38660 whereas the theoretical number of transactions should +have been $200 \cdot 4 \cdot 89 = 71200$. + +At this point, we chose an experiment at the limit of the saturation diagonal, +in order to further study the performance of this release. +**The chosen experiment is (`r=200,c=2`)**. + +This is a plot of the CPU load (average over 1 minute, as output by `top`) of the load runner for (`r=200,c=2`), +where we can see that the load stays close to 0 most of the time. + +![load-load-runner](img34/v034_r200c2_load-runner.png) + +### Examining latencies + +The method described [here](method.md) allows us to plot the latencies of transactions +for all experiments. + +![all-latencies](img34/v034_200node_latencies.png) + +As we can see, even the experiments beyond the saturation diagonal managed to keep +transaction latency stable (i.e. not constantly increasing). +Our interpretation for this is that contention within Tendermint Core was propagated, +via the websockets, to the load runner, +hence the load runner could not produce the target load, but a fraction of it. + +Further examination of the Prometheus data (see below), showed that the mempool contained many transactions +at steady state, but did not grow much without quickly returning to this steady state. This demonstrates +that Tendermint Core network was able to process transactions at least as quickly as they +were submitted to the mempool. Finally, the test script made sure that, at the end of an experiment, the +mempool was empty so that all transactions submitted to the chain were processed. + +Finally, the number of points present in the plot appears to be much less than expected given the +number of transactions in each experiment, particularly close to or above the saturation diagonal. +This is a visual effect of the plot; what appear to be points in the plot are actually potentially huge +clusters of points. To corroborate this, we have zoomed in the plot above by setting (carefully chosen) +tiny axis intervals. The cluster shown below looks like a single point in the plot above. + +![all-latencies-zoomed](img34/v034_200node_latencies_zoomed.png) + +The plot of latencies can we used as a baseline to compare with other releases. + +The following plot summarizes average latencies versus overall throughput +across different numbers of WebSocket connections to the node into which +transactions are being loaded. + +![latency-vs-throughput](img34/v034_latency_throughput.png) + +### Prometheus Metrics on the Chosen Experiment + +As mentioned [above](#finding-the-saturation-point), the chosen experiment is `r=200,c=2`. +This section further examines key metrics for this experiment extracted from Prometheus data. + +#### Mempool Size + +The mempool size, a count of the number of transactions in the mempool, was shown to be stable and homogeneous +at all full nodes. It did not exhibit any unconstrained growth. +The plot below shows the evolution over time of the cumulative number of transactions inside all full nodes' mempools +at a given time. +The two spikes that can be observed correspond to a period where consensus instances proceeded beyond the initial round +at some nodes. + +![mempool-cumulative](img34/v034_r200c2_mempool_size.png) + +The plot below shows evolution of the average over all full nodes, which oscillates between 1500 and 2000 +outstanding transactions. + +![mempool-avg](img34/v034_r200c2_mempool_size_avg.png) + +The peaks observed coincide with the moments when some nodes proceeded beyond the initial round of consensus (see below). + +#### Peers + +The number of peers was stable at all nodes. +It was higher for the seed nodes (around 140) than for the rest (between 21 and 74). +The fact that non-seed nodes reach more than 50 peers is due to #9548. + +![peers](img34/v034_r200c2_peers.png) + +#### Consensus Rounds per Height + +Most nodes used only round 0 for most heights, but some nodes needed to advance to round 1 for some heights. + +![rounds](img34/v034_r200c2_rounds.png) + +#### Blocks Produced per Minute, Transactions Processed per Minute + +The blocks produced per minute are the slope of this plot. + +![heights](img34/v034_r200c2_heights.png) + +Over a period of 2 minutes, the height goes from 530 to 569. +This results in an average of 19.5 blocks produced per minute. + +The transactions processed per minute are the slope of this plot. + +![total-txs](img34/v034_r200c2_total-txs.png) + +Over a period of 2 minutes, the total goes from 64525 to 100125 transactions, +resulting in 17800 transactions per minute. However, we can see in the plot that +all transactions in the load are processed long before the two minutes. +If we adjust the time window when transactions are processed (approx. 105 seconds), +we obtain 20343 transactions per minute. + +#### Memory Resident Set Size + +Resident Set Size of all monitored processes is plotted below. + +![rss](img34/v034_r200c2_rss.png) + +The average over all processes oscillates around 1.2 GiB and does not demonstrate unconstrained growth. + +![rss-avg](img34/v034_r200c2_rss_avg.png) + +#### CPU utilization + +The best metric from Prometheus to gauge CPU utilization in a Unix machine is `load1`, +as it usually appears in the +[output of `top`](https://www.digitalocean.com/community/tutorials/load-average-in-linux). + +![load1](img34/v034_r200c2_load1.png) + +It is contained in most cases below 5, which is generally considered acceptable load. + +### Test Result + +**Result: N/A** (v0.34.x is the baseline) + +Date: 2022-10-14 + +Version: 3ec6e424d6ae4c96867c2dcf8310572156068bb6 + +## Rotating Node Testnet + +For this testnet, we will use a load that can safely be considered below the saturation +point for the size of this testnet (between 13 and 38 full nodes): `c=4,r=800`. + +N.B.: The version of CometBFT used for these tests is affected by #9539. +However, the reduced load that reaches the mempools is orthogonal to functionality +we are focusing on here. + +### Latencies + +The plot of all latencies can be seen in the following plot. + +![rotating-all-latencies](img34/v034_rotating_latencies.png) + +We can observe there are some very high latencies, towards the end of the test. +Upon suspicion that they are duplicate transactions, we examined the latencies +raw file and discovered there are more than 100K duplicate transactions. + +The following plot shows the latencies file where all duplicate transactions have +been removed, i.e., only the first occurrence of a duplicate transaction is kept. + +![rotating-all-latencies-uniq](img34/v034_rotating_latencies_uniq.png) + +This problem, existing in `v0.34.x`, will need to be addressed, perhaps in the same way +we addressed it when running the 200 node test with high loads: increasing the `cache_size` +configuration parameter. + +### Prometheus Metrics + +The set of metrics shown here are less than for the 200 node experiment. +We are only interested in those for which the catch-up process (blocksync) may have an impact. + +#### Blocks and Transactions per minute + +Just as shown for the 200 node test, the blocks produced per minute are the gradient of this plot. + +![rotating-heights](img34/v034_rotating_heights.png) + +Over a period of 5229 seconds, the height goes from 2 to 3638. +This results in an average of 41 blocks produced per minute. + +The following plot shows only the heights reported by ephemeral nodes +(which are also included in the plot above). Note that the _height_ metric +is only showed _once the node has switched to consensus_, hence the gaps +when nodes are killed, wiped out, started from scratch, and catching up. + +![rotating-heights-ephe](img34/v034_rotating_heights_ephe.png) + +The transactions processed per minute are the gradient of this plot. + +![rotating-total-txs](img34/v034_rotating_total-txs.png) + +The small lines we see periodically close to `y=0` are the transactions that +ephemeral nodes start processing when they are caught up. + +Over a period of 5229 minutes, the total goes from 0 to 387697 transactions, +resulting in 4449 transactions per minute. We can see some abrupt changes in +the plot's gradient. This will need to be investigated. + +#### Peers + +The plot below shows the evolution in peers throughout the experiment. +The periodic changes observed are due to the ephemeral nodes being stopped, +wiped out, and recreated. + +![rotating-peers](img34/v034_rotating_peers.png) + +The validators' plots are concentrated at the higher part of the graph, whereas the ephemeral nodes +are mostly at the lower part. + +#### Memory Resident Set Size + +The average Resident Set Size (RSS) over all processes seems stable, and slightly growing toward the end. +This might be related to the increased in transaction load observed above. + +![rotating-rss-avg](img34/v034_rotating_rss_avg.png) + +The memory taken by the validators and the ephemeral nodes (when they are up) is comparable. + +#### CPU utilization + +The plot shows metric `load1` for all nodes. + +![rotating-load1](img34/v034_rotating_load1.png) + +It is contained under 5 most of the time, which is considered normal load. +The purple line, which follows a different pattern is the validator receiving all +transactions, via RPC, from the load runner process. + +### Test Result + +**Result: N/A** + +Date: 2022-10-10 + +Version: a28c987f5a604ff66b515dd415270063e6fb069d diff --git a/cometbft/v0.38/docs/qa/TMCore-QA-37.mdx b/cometbft/v0.38/docs/qa/TMCore-QA-37.mdx new file mode 100644 index 00000000..23dd2ed1 --- /dev/null +++ b/cometbft/v0.38/docs/qa/TMCore-QA-37.mdx @@ -0,0 +1,324 @@ +--- +order: 1 +parent: + title: Tendermint Core QA Results v0.37.x + description: This is a report on the results obtained when running TM v0.37.x on testnets + order: 4 +--- + +# Tendermint Core QA Results v0.37.x + +## Issues discovered + +During this iteration of the QA process, the following issues were found: + +* (critical, fixed) [\#9533] - This bug caused full nodes to sometimes get stuck + when blocksyncing, requiring a manual restart to unblock them. Importantly, + this bug was also present in v0.34.x and the fix was also backported in + [\#9534]. +* (critical, fixed) [\#9539] - `loadtime` is very likely to include more than + one "=" character in transactions, with is rejected by the e2e application. +* (critical, fixed) [\#9581] - Absent prometheus label makes CometBFT crash + when enabling Prometheus metric collection +* (non-critical, not fixed) [\#9548] - Full nodes can go over 50 connected + peers, which is not intended by the default configuration. +* (non-critical, not fixed) [\#9537] - With the default mempool cache setting, + duplicated transactions are not rejected when gossipped and eventually flood + all mempools. The 200 node testnets were thus run with a value of 200000 (as + opposed to the default 10000) + +## 200 Node Testnet + +### Finding the Saturation Point + +The first goal is to identify the saturation point and compare it with the baseline (v0.34.x). +For further details, see [this paragraph](TMCore-QA-34.md#finding-the-saturation-point) +in the baseline version. + +The following table summarizes the results for v0.37.x, for the different experiments +(extracted from file [`v037_report_tabbed.txt`](img37/200nodes_tm037/v037_report_tabbed.txt)). + +The X axis of this table is `c`, the number of connections created by the load runner process to the target node. +The Y axis of this table is `r`, the rate or number of transactions issued per second. + +| | c=1 | c=2 | c=4 | +| :--- | ----: | ----: | ----: | +| r=25 | 2225 | 4450 | 8900 | +| r=50 | 4450 | 8900 | 17800 | +| r=100 | 8900 | 17800 | 35600 | +| r=200 | 17800 | 35600 | 38660 | + +For comparison, this is the table with the baseline version. + +| | c=1 | c=2 | c=4 | +| :--- | ----: | ----: | ----: | +| r=25 | 2225 | 4450 | 8900 | +| r=50 | 4450 | 8900 | 17800 | +| r=100 | 8900 | 17800 | 35400 | +| r=200 | 17800 | 35600 | 37358 | + +The saturation point is beyond the diagonal: + +* `r=200,c=2` +* `r=100,c=4` + +which is at the same place as the baseline. For more details on the saturation point, see +[this paragraph](TMCore-QA-34.md#finding-the-saturation-point) in the baseline version. + +The experiment chosen to examine Prometheus metrics is the same as in the baseline: +**`r=200,c=2`**. + +The load runner's CPU load was negligible (near 0) when running `r=200,c=2`. + +### Examining latencies + +The method described [here](method.md) allows us to plot the latencies of transactions +for all experiments. + +![all-latencies](img37/200nodes_tm037/v037_200node_latencies.png) + +The data seen in the plot is similar to that of the baseline. + +![all-latencies-bl](img34/v034_200node_latencies.png) + +Therefore, for further details on these plots, +see [this paragraph](CometBFT-QA-34.md#examining-latencies) in the baseline version. + +The following plot summarizes average latencies versus overall throughputs +across different numbers of WebSocket connections to the node into which +transactions are being loaded. + +![latency-vs-throughput](img37/200nodes_tm037/v037_latency_throughput.png) + +This is similar to that of the baseline plot: + +![latency-vs-throughput-bl](img34/v034_latency_throughput.png) + +### Prometheus Metrics on the Chosen Experiment + +As mentioned [above](#finding-the-saturation-point), the chosen experiment is `r=200,c=2`. +This section further examines key metrics for this experiment extracted from Prometheus data. + +#### Mempool Size + +The mempool size, a count of the number of transactions in the mempool, was shown to be stable and homogeneous +at all full nodes. It did not exhibit any unconstrained growth. +The plot below shows the evolution over time of the cumulative number of transactions inside all full nodes' mempools +at a given time. + +![mempool-cumulative](img37/200nodes_tm037/v037_r200c2_mempool_size.png) + +The plot below shows evolution of the average over all full nodes, which oscillate between 1500 and 2000 outstanding transactions. + +![mempool-avg](img37/200nodes_tm037/v037_r200c2_mempool_size_avg.png) + +The peaks observed coincide with the moments when some nodes reached round 1 of consensus (see below). + +**These plots yield similar results to the baseline**: + +![mempool-cumulative-bl](img34/v034_r200c2_mempool_size.png) + +![mempool-avg-bl](img34/v034_r200c2_mempool_size_avg.png) + +#### Peers + +The number of peers was stable at all nodes. +It was higher for the seed nodes (around 140) than for the rest (between 16 and 78). + +![peers](img37/200nodes_tm037/v037_r200c2_peers.png) + +Just as in the baseline, the fact that non-seed nodes reach more than 50 peers is due to #9548. + +**This plot yields similar results to the baseline**: + +![peers-bl](img34/v034_r200c2_peers.png) + +#### Consensus Rounds per Height + +Most heights took just one round, but some nodes needed to advance to round 1 at some point. + +![rounds](img37/200nodes_tm037/v037_r200c2_rounds.png) + +**This plot yields slightly better results than the baseline**: + +![rounds-bl](img34/v034_r200c2_rounds.png) + +#### Blocks Produced per Minute, Transactions Processed per Minute + +The blocks produced per minute are the gradient of this plot. + +![heights](img37/200nodes_tm037/v037_r200c2_heights.png) + +Over a period of 2 minutes, the height goes from 477 to 524. +This results in an average of 23.5 blocks produced per minute. + +The transactions processed per minute are the gradient of this plot. + +![total-txs](img37/200nodes_tm037/v037_r200c2_total-txs.png) + +Over a period of 2 minutes, the total goes from 64525 to 100125 transactions, +resulting in 17800 transactions per minute. However, we can see in the plot that +all transactions in the load are process long before the two minutes. +If we adjust the time window when transactions are processed (approx. 90 seconds), +we obtain 23733 transactions per minute. + +**These plots yield similar results to the baseline**: + +![heights-bl](img34/v034_r200c2_heights.png) + +![total-txs](img34/v034_r200c2_total-txs.png) + +#### Memory Resident Set Size + +Resident Set Size of all monitored processes is plotted below. + +![rss](img37/200nodes_tm037/v037_r200c2_rss.png) + +The average over all processes oscillates around 380 MiB and does not demonstrate unconstrained growth. + +![rss-avg](img37/200nodes_tm037/v037_r200c2_rss_avg.png) + +**These plots yield similar results to the baseline**: + +![rss-bl](img34/v034_r200c2_rss.png) + +![rss-avg-bl](img34/v034_r200c2_rss_avg.png) + +#### CPU utilization + +The best metric from Prometheus to gauge CPU utilization in a Unix machine is `load1`, +as it usually appears in the +[output of `top`](https://www.digitalocean.com/community/tutorials/load-average-in-linux). + +![load1](img37/200nodes_tm037/v037_r200c2_load1.png) + +It is contained below 5 on most nodes. + +**This plot yields similar results to the baseline**: + +![load1](img34/v034_r200c2_load1.png) + +### Test Result + +**Result: PASS** + +Date: 2022-10-14 + +Version: 1cf9d8e276afe8595cba960b51cd056514965fd1 + +## Rotating Node Testnet + +We use the same load as in the baseline: `c=4,r=800`. + +Just as in the baseline tests, the version of CometBFT used for these tests is affected by #9539. +See this paragraph in the [baseline report](method.md#rotating-node-testnet) for further details. +Finally, note that this setup allows for a fairer comparison between this version and the baseline. + +### Latencies + +The plot of all latencies can be seen here. + +![rotating-all-latencies](img37/200nodes_tm037/v037_rotating_latencies.png) + +Which is similar to the baseline. + +![rotating-all-latencies-bl](img34/v034_rotating_latencies_uniq.png) + +Note that we are comparing against the baseline plot with _unique_ +transactions. This is because the problem with duplicate transactions +detected during the baseline experiment did not show up for `v0.37`, +which is _not_ proof that the problem is not present in `v0.37`. + +### Prometheus Metrics + +The set of metrics shown here match those shown on the baseline (`v0.34`) for the same experiment. +We also show the baseline results for comparison. + +#### Blocks and Transactions per minute + +The blocks produced per minute are the gradient of this plot. + +![rotating-heights](img37/200nodes_tm037/v037_rotating_heights.png) + +Over a period of 4446 seconds, the height goes from 5 to 3323. +This results in an average of 45 blocks produced per minute, +which is similar to the baseline, shown below. + +![rotating-heights-bl](img34/v034_rotating_heights.png) + +The following two plots show only the heights reported by ephemeral nodes. +The second plot is the baseline plot for comparison. + +![rotating-heights-ephe](img37/200nodes_tm037/v037_rotating_heights_ephe.png) + +![rotating-heights-ephe-bl](img34/v034_rotating_heights_ephe.png) + +By the length of the segments, we can see that ephemeral nodes in `v0.37` +catch up slightly faster. + +The transactions processed per minute are the gradient of this plot. + +![rotating-total-txs](img37/200nodes_tm037/v037_rotating_total-txs.png) + +Over a period of 3852 seconds, the total goes from 597 to 267298 transactions in one of the validators, +resulting in 4154 transactions per minute, which is slightly lower than the baseline, +although the baseline had to deal with duplicate transactions. + +For comparison, this is the baseline plot. + +![rotating-total-txs-bl](img34/v034_rotating_total-txs.png) + +#### Peers + +The plot below shows the evolution of the number of peers throughout the experiment. + +![rotating-peers](img37/200nodes_tm037/v037_rotating_peers.png) + +This is the baseline plot, for comparison. + +![rotating-peers-bl](img34/v034_rotating_peers.png) + +The plotted values and their evolution are comparable in both plots. + +For further details on these plots, see the baseline report. + +#### Memory Resident Set Size + +The average Resident Set Size (RSS) over all processes looks slightly more stable +on `v0.37` (first plot) than on the baseline (second plot). + +![rotating-rss-avg](img37/200nodes_tm037/v037_rotating_rss_avg.png) + +![rotating-rss-avg-bl](img34/v034_rotating_rss_avg.png) + +The memory taken by the validators and the ephemeral nodes when they are up is comparable (not shown in the plots), +just as observed in the baseline. + +#### CPU utilization + +The plot shows metric `load1` for all nodes. + +![rotating-load1](img37/200nodes_tm037/v037_rotating_load1.png) + +![rotating-load1-bl](img34/v034_rotating_load1.png) + +In both cases, it is contained under 5 most of the time, which is considered normal load. +The green line in the `v0.37` plot and the purple line in the baseline plot (`v0.34`) +correspond to the validators receiving all transactions, via RPC, from the load runner process. +In both cases, they oscillate around 5 (normal load). The main difference is that other +nodes are generally less loaded in `v0.37`. + +### Test Result + +**Result: PASS** + +Date: 2022-10-10 + +Version: 155110007b9d8b83997a799016c1d0844c8efbaf + +[\#9533]: https://github.com/tendermint/tendermint/pull/9533 +[\#9534]: https://github.com/tendermint/tendermint/pull/9534 +[\#9539]: https://github.com/tendermint/tendermint/issues/9539 +[\#9548]: https://github.com/tendermint/tendermint/issues/9548 +[\#9537]: https://github.com/tendermint/tendermint/issues/9537 +[\#9581]: https://github.com/tendermint/tendermint/issues/9581 diff --git a/cometbft/v0.38/docs/qa/img34/baseline/avg_cpu.png b/cometbft/v0.38/docs/qa/img34/baseline/avg_cpu.png new file mode 100644 index 00000000..622456df Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/baseline/avg_cpu.png differ diff --git a/cometbft/v0.38/docs/qa/img34/baseline/avg_memory.png b/cometbft/v0.38/docs/qa/img34/baseline/avg_memory.png new file mode 100644 index 00000000..55f213f5 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/baseline/avg_memory.png differ diff --git a/cometbft/v0.38/docs/qa/img34/baseline/avg_mempool_size.png b/cometbft/v0.38/docs/qa/img34/baseline/avg_mempool_size.png new file mode 100644 index 00000000..ec740729 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/baseline/avg_mempool_size.png differ diff --git a/cometbft/v0.38/docs/qa/img34/baseline/block_rate_regular.png b/cometbft/v0.38/docs/qa/img34/baseline/block_rate_regular.png new file mode 100644 index 00000000..bdc7aa28 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/baseline/block_rate_regular.png differ diff --git a/cometbft/v0.38/docs/qa/img34/baseline/cpu.png b/cometbft/v0.38/docs/qa/img34/baseline/cpu.png new file mode 100644 index 00000000..ac4fc269 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/baseline/cpu.png differ diff --git a/cometbft/v0.38/docs/qa/img34/baseline/memory.png b/cometbft/v0.38/docs/qa/img34/baseline/memory.png new file mode 100644 index 00000000..17336bd1 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/baseline/memory.png differ diff --git a/cometbft/v0.38/docs/qa/img34/baseline/mempool_size.png b/cometbft/v0.38/docs/qa/img34/baseline/mempool_size.png new file mode 100644 index 00000000..fafba68c Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/baseline/mempool_size.png differ diff --git a/cometbft/v0.38/docs/qa/img34/baseline/peers.png b/cometbft/v0.38/docs/qa/img34/baseline/peers.png new file mode 100644 index 00000000..05a288a3 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/baseline/peers.png differ diff --git a/cometbft/v0.38/docs/qa/img34/baseline/rounds.png b/cometbft/v0.38/docs/qa/img34/baseline/rounds.png new file mode 100644 index 00000000..79f3348a Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/baseline/rounds.png differ diff --git a/cometbft/v0.38/docs/qa/img34/baseline/total_txs_rate_regular.png b/cometbft/v0.38/docs/qa/img34/baseline/total_txs_rate_regular.png new file mode 100644 index 00000000..d80bef12 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/baseline/total_txs_rate_regular.png differ diff --git a/cometbft/v0.38/docs/qa/img34/cmt1tm1/all_experiments.png b/cometbft/v0.38/docs/qa/img34/cmt1tm1/all_experiments.png new file mode 100644 index 00000000..4dc857ed Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/cmt1tm1/all_experiments.png differ diff --git a/cometbft/v0.38/docs/qa/img34/cmt1tm1/avg_cpu.png b/cometbft/v0.38/docs/qa/img34/cmt1tm1/avg_cpu.png new file mode 100644 index 00000000..cabd273a Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/cmt1tm1/avg_cpu.png differ diff --git a/cometbft/v0.38/docs/qa/img34/cmt1tm1/avg_memory.png b/cometbft/v0.38/docs/qa/img34/cmt1tm1/avg_memory.png new file mode 100644 index 00000000..c8e57617 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/cmt1tm1/avg_memory.png differ diff --git a/cometbft/v0.38/docs/qa/img34/cmt1tm1/avg_mempool_size.png b/cometbft/v0.38/docs/qa/img34/cmt1tm1/avg_mempool_size.png new file mode 100644 index 00000000..b41199dc Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/cmt1tm1/avg_mempool_size.png differ diff --git a/cometbft/v0.38/docs/qa/img34/cmt1tm1/block_rate_regular.png b/cometbft/v0.38/docs/qa/img34/cmt1tm1/block_rate_regular.png new file mode 100644 index 00000000..9b3a0b82 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/cmt1tm1/block_rate_regular.png differ diff --git a/cometbft/v0.38/docs/qa/img34/cmt1tm1/cpu.png b/cometbft/v0.38/docs/qa/img34/cmt1tm1/cpu.png new file mode 100644 index 00000000..cd5acdeb Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/cmt1tm1/cpu.png differ diff --git a/cometbft/v0.38/docs/qa/img34/cmt1tm1/memory.png b/cometbft/v0.38/docs/qa/img34/cmt1tm1/memory.png new file mode 100644 index 00000000..6f56b3cc Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/cmt1tm1/memory.png differ diff --git a/cometbft/v0.38/docs/qa/img34/cmt1tm1/mempool_size.png b/cometbft/v0.38/docs/qa/img34/cmt1tm1/mempool_size.png new file mode 100644 index 00000000..862a0bdd Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/cmt1tm1/mempool_size.png differ diff --git a/cometbft/v0.38/docs/qa/img34/cmt1tm1/peers.png b/cometbft/v0.38/docs/qa/img34/cmt1tm1/peers.png new file mode 100644 index 00000000..737cf3df Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/cmt1tm1/peers.png differ diff --git a/cometbft/v0.38/docs/qa/img34/cmt1tm1/rounds.png b/cometbft/v0.38/docs/qa/img34/cmt1tm1/rounds.png new file mode 100644 index 00000000..17884813 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/cmt1tm1/rounds.png differ diff --git a/cometbft/v0.38/docs/qa/img34/cmt1tm1/total_txs_rate_regular.png b/cometbft/v0.38/docs/qa/img34/cmt1tm1/total_txs_rate_regular.png new file mode 100644 index 00000000..8b0cc0d4 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/cmt1tm1/total_txs_rate_regular.png differ diff --git a/cometbft/v0.38/docs/qa/img34/cmt2tm1/all_experiments.png b/cometbft/v0.38/docs/qa/img34/cmt2tm1/all_experiments.png new file mode 100644 index 00000000..4e6f73d3 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/cmt2tm1/all_experiments.png differ diff --git a/cometbft/v0.38/docs/qa/img34/cmt2tm1/avg_cpu.png b/cometbft/v0.38/docs/qa/img34/cmt2tm1/avg_cpu.png new file mode 100644 index 00000000..92fea31b Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/cmt2tm1/avg_cpu.png differ diff --git a/cometbft/v0.38/docs/qa/img34/cmt2tm1/avg_memory.png b/cometbft/v0.38/docs/qa/img34/cmt2tm1/avg_memory.png new file mode 100644 index 00000000..f362798d Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/cmt2tm1/avg_memory.png differ diff --git a/cometbft/v0.38/docs/qa/img34/cmt2tm1/avg_mempool_size.png b/cometbft/v0.38/docs/qa/img34/cmt2tm1/avg_mempool_size.png new file mode 100644 index 00000000..b73e577b Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/cmt2tm1/avg_mempool_size.png differ diff --git a/cometbft/v0.38/docs/qa/img34/cmt2tm1/block_rate_regular.png b/cometbft/v0.38/docs/qa/img34/cmt2tm1/block_rate_regular.png new file mode 100644 index 00000000..5fc7a556 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/cmt2tm1/block_rate_regular.png differ diff --git a/cometbft/v0.38/docs/qa/img34/cmt2tm1/cpu.png b/cometbft/v0.38/docs/qa/img34/cmt2tm1/cpu.png new file mode 100644 index 00000000..15df58ab Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/cmt2tm1/cpu.png differ diff --git a/cometbft/v0.38/docs/qa/img34/cmt2tm1/memory.png b/cometbft/v0.38/docs/qa/img34/cmt2tm1/memory.png new file mode 100644 index 00000000..b0feab10 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/cmt2tm1/memory.png differ diff --git a/cometbft/v0.38/docs/qa/img34/cmt2tm1/mempool_size.png b/cometbft/v0.38/docs/qa/img34/cmt2tm1/mempool_size.png new file mode 100644 index 00000000..b3a1514f Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/cmt2tm1/mempool_size.png differ diff --git a/cometbft/v0.38/docs/qa/img34/cmt2tm1/peers.png b/cometbft/v0.38/docs/qa/img34/cmt2tm1/peers.png new file mode 100644 index 00000000..558d4c12 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/cmt2tm1/peers.png differ diff --git a/cometbft/v0.38/docs/qa/img34/cmt2tm1/rounds.png b/cometbft/v0.38/docs/qa/img34/cmt2tm1/rounds.png new file mode 100644 index 00000000..3c22a5cf Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/cmt2tm1/rounds.png differ diff --git a/cometbft/v0.38/docs/qa/img34/cmt2tm1/total_txs_rate_regular.png b/cometbft/v0.38/docs/qa/img34/cmt2tm1/total_txs_rate_regular.png new file mode 100644 index 00000000..ae98df21 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/cmt2tm1/total_txs_rate_regular.png differ diff --git a/cometbft/v0.38/docs/qa/img34/homogeneous/all_experiments.png b/cometbft/v0.38/docs/qa/img34/homogeneous/all_experiments.png new file mode 100644 index 00000000..d8768f6a Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/homogeneous/all_experiments.png differ diff --git a/cometbft/v0.38/docs/qa/img34/homogeneous/avg_cpu.png b/cometbft/v0.38/docs/qa/img34/homogeneous/avg_cpu.png new file mode 100644 index 00000000..7df18895 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/homogeneous/avg_cpu.png differ diff --git a/cometbft/v0.38/docs/qa/img34/homogeneous/avg_memory.png b/cometbft/v0.38/docs/qa/img34/homogeneous/avg_memory.png new file mode 100644 index 00000000..e800cbce Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/homogeneous/avg_memory.png differ diff --git a/cometbft/v0.38/docs/qa/img34/homogeneous/avg_mempool_size.png b/cometbft/v0.38/docs/qa/img34/homogeneous/avg_mempool_size.png new file mode 100644 index 00000000..beb323e6 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/homogeneous/avg_mempool_size.png differ diff --git a/cometbft/v0.38/docs/qa/img34/homogeneous/block_rate_regular.png b/cometbft/v0.38/docs/qa/img34/homogeneous/block_rate_regular.png new file mode 100644 index 00000000..2a71ab70 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/homogeneous/block_rate_regular.png differ diff --git a/cometbft/v0.38/docs/qa/img34/homogeneous/cpu.png b/cometbft/v0.38/docs/qa/img34/homogeneous/cpu.png new file mode 100644 index 00000000..8e8c9227 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/homogeneous/cpu.png differ diff --git a/cometbft/v0.38/docs/qa/img34/homogeneous/memory.png b/cometbft/v0.38/docs/qa/img34/homogeneous/memory.png new file mode 100644 index 00000000..190c622a Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/homogeneous/memory.png differ diff --git a/cometbft/v0.38/docs/qa/img34/homogeneous/mempool_size.png b/cometbft/v0.38/docs/qa/img34/homogeneous/mempool_size.png new file mode 100644 index 00000000..ec1c79a2 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/homogeneous/mempool_size.png differ diff --git a/cometbft/v0.38/docs/qa/img34/homogeneous/peers.png b/cometbft/v0.38/docs/qa/img34/homogeneous/peers.png new file mode 100644 index 00000000..3c8b0a2e Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/homogeneous/peers.png differ diff --git a/cometbft/v0.38/docs/qa/img34/homogeneous/rounds.png b/cometbft/v0.38/docs/qa/img34/homogeneous/rounds.png new file mode 100644 index 00000000..660f31d9 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/homogeneous/rounds.png differ diff --git a/cometbft/v0.38/docs/qa/img34/homogeneous/total_txs_rate_regular.png b/cometbft/v0.38/docs/qa/img34/homogeneous/total_txs_rate_regular.png new file mode 100644 index 00000000..a9025b66 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/homogeneous/total_txs_rate_regular.png differ diff --git a/cometbft/v0.38/docs/qa/img34/v034_200node_latencies.png b/cometbft/v0.38/docs/qa/img34/v034_200node_latencies.png new file mode 100644 index 00000000..afd1060c Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/v034_200node_latencies.png differ diff --git a/cometbft/v0.38/docs/qa/img34/v034_200node_latencies_zoomed.png b/cometbft/v0.38/docs/qa/img34/v034_200node_latencies_zoomed.png new file mode 100644 index 00000000..1ff93644 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/v034_200node_latencies_zoomed.png differ diff --git a/cometbft/v0.38/docs/qa/img34/v034_200node_tm2cmt1/all_experiments.png b/cometbft/v0.38/docs/qa/img34/v034_200node_tm2cmt1/all_experiments.png new file mode 100644 index 00000000..e91a87ef Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/v034_200node_tm2cmt1/all_experiments.png differ diff --git a/cometbft/v0.38/docs/qa/img34/v034_200node_tm2cmt1/avg_cpu.png b/cometbft/v0.38/docs/qa/img34/v034_200node_tm2cmt1/avg_cpu.png new file mode 100644 index 00000000..a1b0ef79 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/v034_200node_tm2cmt1/avg_cpu.png differ diff --git a/cometbft/v0.38/docs/qa/img34/v034_200node_tm2cmt1/avg_memory.png b/cometbft/v0.38/docs/qa/img34/v034_200node_tm2cmt1/avg_memory.png new file mode 100644 index 00000000..f9d9b993 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/v034_200node_tm2cmt1/avg_memory.png differ diff --git a/cometbft/v0.38/docs/qa/img34/v034_200node_tm2cmt1/avg_mempool_size.png b/cometbft/v0.38/docs/qa/img34/v034_200node_tm2cmt1/avg_mempool_size.png new file mode 100644 index 00000000..c2b89606 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/v034_200node_tm2cmt1/avg_mempool_size.png differ diff --git a/cometbft/v0.38/docs/qa/img34/v034_200node_tm2cmt1/block_rate_regular.png b/cometbft/v0.38/docs/qa/img34/v034_200node_tm2cmt1/block_rate_regular.png new file mode 100644 index 00000000..5a5417bd Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/v034_200node_tm2cmt1/block_rate_regular.png differ diff --git a/cometbft/v0.38/docs/qa/img34/v034_200node_tm2cmt1/c2r200_merged.png b/cometbft/v0.38/docs/qa/img34/v034_200node_tm2cmt1/c2r200_merged.png new file mode 100644 index 00000000..45de9ce7 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/v034_200node_tm2cmt1/c2r200_merged.png differ diff --git a/cometbft/v0.38/docs/qa/img34/v034_200node_tm2cmt1/cpu.png b/cometbft/v0.38/docs/qa/img34/v034_200node_tm2cmt1/cpu.png new file mode 100644 index 00000000..eabfa966 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/v034_200node_tm2cmt1/cpu.png differ diff --git a/cometbft/v0.38/docs/qa/img34/v034_200node_tm2cmt1/memory.png b/cometbft/v0.38/docs/qa/img34/v034_200node_tm2cmt1/memory.png new file mode 100644 index 00000000..70014c1f Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/v034_200node_tm2cmt1/memory.png differ diff --git a/cometbft/v0.38/docs/qa/img34/v034_200node_tm2cmt1/mempool_size.png b/cometbft/v0.38/docs/qa/img34/v034_200node_tm2cmt1/mempool_size.png new file mode 100644 index 00000000..5f4c44b2 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/v034_200node_tm2cmt1/mempool_size.png differ diff --git a/cometbft/v0.38/docs/qa/img34/v034_200node_tm2cmt1/peers.png b/cometbft/v0.38/docs/qa/img34/v034_200node_tm2cmt1/peers.png new file mode 100644 index 00000000..c35c8467 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/v034_200node_tm2cmt1/peers.png differ diff --git a/cometbft/v0.38/docs/qa/img34/v034_200node_tm2cmt1/rounds.png b/cometbft/v0.38/docs/qa/img34/v034_200node_tm2cmt1/rounds.png new file mode 100644 index 00000000..7d1034bc Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/v034_200node_tm2cmt1/rounds.png differ diff --git a/cometbft/v0.38/docs/qa/img34/v034_200node_tm2cmt1/total_txs_rate_regular.png b/cometbft/v0.38/docs/qa/img34/v034_200node_tm2cmt1/total_txs_rate_regular.png new file mode 100644 index 00000000..2e8a40af Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/v034_200node_tm2cmt1/total_txs_rate_regular.png differ diff --git a/cometbft/v0.38/docs/qa/img34/v034_latency_throughput.png b/cometbft/v0.38/docs/qa/img34/v034_latency_throughput.png new file mode 100644 index 00000000..3674fe47 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/v034_latency_throughput.png differ diff --git a/cometbft/v0.38/docs/qa/img34/v034_r200c2_heights.png b/cometbft/v0.38/docs/qa/img34/v034_r200c2_heights.png new file mode 100644 index 00000000..11f3bba4 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/v034_r200c2_heights.png differ diff --git a/cometbft/v0.38/docs/qa/img34/v034_r200c2_load-runner.png b/cometbft/v0.38/docs/qa/img34/v034_r200c2_load-runner.png new file mode 100644 index 00000000..70211b0d Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/v034_r200c2_load-runner.png differ diff --git a/cometbft/v0.38/docs/qa/img34/v034_r200c2_load1.png b/cometbft/v0.38/docs/qa/img34/v034_r200c2_load1.png new file mode 100644 index 00000000..11012844 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/v034_r200c2_load1.png differ diff --git a/cometbft/v0.38/docs/qa/img34/v034_r200c2_mempool_size.png b/cometbft/v0.38/docs/qa/img34/v034_r200c2_mempool_size.png new file mode 100644 index 00000000..c5d69020 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/v034_r200c2_mempool_size.png differ diff --git a/cometbft/v0.38/docs/qa/img34/v034_r200c2_mempool_size_avg.png b/cometbft/v0.38/docs/qa/img34/v034_r200c2_mempool_size_avg.png new file mode 100644 index 00000000..bda399fe Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/v034_r200c2_mempool_size_avg.png differ diff --git a/cometbft/v0.38/docs/qa/img34/v034_r200c2_peers.png b/cometbft/v0.38/docs/qa/img34/v034_r200c2_peers.png new file mode 100644 index 00000000..a0aea7ad Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/v034_r200c2_peers.png differ diff --git a/cometbft/v0.38/docs/qa/img34/v034_r200c2_rounds.png b/cometbft/v0.38/docs/qa/img34/v034_r200c2_rounds.png new file mode 100644 index 00000000..215be100 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/v034_r200c2_rounds.png differ diff --git a/cometbft/v0.38/docs/qa/img34/v034_r200c2_rss.png b/cometbft/v0.38/docs/qa/img34/v034_r200c2_rss.png new file mode 100644 index 00000000..6d14dced Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/v034_r200c2_rss.png differ diff --git a/cometbft/v0.38/docs/qa/img34/v034_r200c2_rss_avg.png b/cometbft/v0.38/docs/qa/img34/v034_r200c2_rss_avg.png new file mode 100644 index 00000000..8dec67da Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/v034_r200c2_rss_avg.png differ diff --git a/cometbft/v0.38/docs/qa/img34/v034_r200c2_total-txs.png b/cometbft/v0.38/docs/qa/img34/v034_r200c2_total-txs.png new file mode 100644 index 00000000..177d5f1c Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/v034_r200c2_total-txs.png differ diff --git a/cometbft/v0.38/docs/qa/img34/v034_report_tabbed.txt b/cometbft/v0.38/docs/qa/img34/v034_report_tabbed.txt new file mode 100644 index 00000000..25149547 --- /dev/null +++ b/cometbft/v0.38/docs/qa/img34/v034_report_tabbed.txt @@ -0,0 +1,52 @@ +Experiment ID: 3d5cf4ef-1a1a-4b46-aa2d-da5643d2e81e │Experiment ID: 80e472ec-13a1-4772-a827-3b0c907fb51d │Experiment ID: 07aca6cf-c5a4-4696-988f-e3270fc6333b + │ │ + Connections: 1 │ Connections: 2 │ Connections: 4 + Rate: 25 │ Rate: 25 │ Rate: 25 + Size: 1024 │ Size: 1024 │ Size: 1024 + │ │ + Total Valid Tx: 2225 │ Total Valid Tx: 4450 │ Total Valid Tx: 8900 + Total Negative Latencies: 0 │ Total Negative Latencies: 0 │ Total Negative Latencies: 0 + Minimum Latency: 599.404362ms │ Minimum Latency: 448.145181ms │ Minimum Latency: 412.485729ms + Maximum Latency: 3.539686885s │ Maximum Latency: 3.237392049s │ Maximum Latency: 12.026665368s + Average Latency: 1.441485349s │ Average Latency: 1.441267946s │ Average Latency: 2.150192457s + Standard Deviation: 541.049869ms │ Standard Deviation: 525.040007ms │ Standard Deviation: 2.233852478s + │ │ +Experiment ID: 953dc544-dd40-40e8-8712-20c34c3ce45e │Experiment ID: d31fc258-16e7-45cd-9dc8-13ab87bc0b0a │Experiment ID: 15d90a7e-b941-42f4-b411-2f15f857739e + │ │ + Connections: 1 │ Connections: 2 │ Connections: 4 + Rate: 50 │ Rate: 50 │ Rate: 50 + Size: 1024 │ Size: 1024 │ Size: 1024 + │ │ + Total Valid Tx: 4450 │ Total Valid Tx: 8900 │ Total Valid Tx: 17800 + Total Negative Latencies: 0 │ Total Negative Latencies: 0 │ Total Negative Latencies: 0 + Minimum Latency: 482.046942ms │ Minimum Latency: 435.458913ms │ Minimum Latency: 510.746448ms + Maximum Latency: 3.761483455s │ Maximum Latency: 7.175583584s │ Maximum Latency: 6.551497882s + Average Latency: 1.450408183s │ Average Latency: 1.681673116s │ Average Latency: 1.738083875s + Standard Deviation: 587.560056ms │ Standard Deviation: 1.147902047s │ Standard Deviation: 943.46522ms + │ │ +Experiment ID: 9a0b9980-9ce6-4db5-a80a-65ca70294b87 │Experiment ID: df8fa4f4-80af-4ded-8a28-356d15018b43 │Experiment ID: d0e41c2c-89c0-4f38-8e34-ca07adae593a + │ │ + Connections: 1 │ Connections: 2 │ Connections: 4 + Rate: 100 │ Rate: 100 │ Rate: 100 + Size: 1024 │ Size: 1024 │ Size: 1024 + │ │ + Total Valid Tx: 8900 │ Total Valid Tx: 17800 │ Total Valid Tx: 35600 + Total Negative Latencies: 0 │ Total Negative Latencies: 0 │ Total Negative Latencies: 0 + Minimum Latency: 477.417219ms │ Minimum Latency: 564.29247ms │ Minimum Latency: 840.71089ms + Maximum Latency: 6.63744785s │ Maximum Latency: 6.988553219s │ Maximum Latency: 9.555312398s + Average Latency: 1.561216103s │ Average Latency: 1.76419063s │ Average Latency: 3.200941683s + Standard Deviation: 1.011333552s │ Standard Deviation: 1.068459423s │ Standard Deviation: 1.732346601s + │ │ +Experiment ID: 493df3ee-4a36-4bce-80f8-6d65da66beda │Experiment ID: 13060525-f04f-46f6-8ade-286684b2fe50 │Experiment ID: 1777cbd2-8c96-42e4-9ec7-9b21f2225e4d + │ │ + Connections: 1 │ Connections: 2 │ Connections: 4 + Rate: 200 │ Rate: 200 │ Rate: 200 + Size: 1024 │ Size: 1024 │ Size: 1024 + │ │ + Total Valid Tx: 17800 │ Total Valid Tx: 35600 │ Total Valid Tx: 38660 + Total Negative Latencies: 0 │ Total Negative Latencies: 0 │ Total Negative Latencies: 0 + Minimum Latency: 493.705261ms │ Minimum Latency: 955.090573ms │ Minimum Latency: 1.9485821s + Maximum Latency: 7.440921872s │ Maximum Latency: 10.086673491s │ Maximum Latency: 17.73103976s + Average Latency: 1.875510582s │ Average Latency: 3.438130099s │ Average Latency: 8.143862237s + Standard Deviation: 1.304336995s │ Standard Deviation: 1.966391574s │ Standard Deviation: 3.943140002s + diff --git a/cometbft/v0.38/docs/qa/img34/v034_rotating_heights.png b/cometbft/v0.38/docs/qa/img34/v034_rotating_heights.png new file mode 100644 index 00000000..47913c28 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/v034_rotating_heights.png differ diff --git a/cometbft/v0.38/docs/qa/img34/v034_rotating_heights_ephe.png b/cometbft/v0.38/docs/qa/img34/v034_rotating_heights_ephe.png new file mode 100644 index 00000000..981b93d6 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/v034_rotating_heights_ephe.png differ diff --git a/cometbft/v0.38/docs/qa/img34/v034_rotating_latencies.png b/cometbft/v0.38/docs/qa/img34/v034_rotating_latencies.png new file mode 100644 index 00000000..f0a54ed5 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/v034_rotating_latencies.png differ diff --git a/cometbft/v0.38/docs/qa/img34/v034_rotating_latencies_uniq.png b/cometbft/v0.38/docs/qa/img34/v034_rotating_latencies_uniq.png new file mode 100644 index 00000000..e5d694a1 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/v034_rotating_latencies_uniq.png differ diff --git a/cometbft/v0.38/docs/qa/img34/v034_rotating_load1.png b/cometbft/v0.38/docs/qa/img34/v034_rotating_load1.png new file mode 100644 index 00000000..e9c385b8 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/v034_rotating_load1.png differ diff --git a/cometbft/v0.38/docs/qa/img34/v034_rotating_peers.png b/cometbft/v0.38/docs/qa/img34/v034_rotating_peers.png new file mode 100644 index 00000000..ab5c8732 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/v034_rotating_peers.png differ diff --git a/cometbft/v0.38/docs/qa/img34/v034_rotating_rss_avg.png b/cometbft/v0.38/docs/qa/img34/v034_rotating_rss_avg.png new file mode 100644 index 00000000..9a416732 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/v034_rotating_rss_avg.png differ diff --git a/cometbft/v0.38/docs/qa/img34/v034_rotating_total-txs.png b/cometbft/v0.38/docs/qa/img34/v034_rotating_total-txs.png new file mode 100644 index 00000000..1ce5f47e Binary files /dev/null and b/cometbft/v0.38/docs/qa/img34/v034_rotating_total-txs.png differ diff --git a/cometbft/v0.38/docs/qa/img37/200nodes_cmt037/all_experiments.png b/cometbft/v0.38/docs/qa/img37/200nodes_cmt037/all_experiments.png new file mode 100644 index 00000000..a6189610 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img37/200nodes_cmt037/all_experiments.png differ diff --git a/cometbft/v0.38/docs/qa/img37/200nodes_cmt037/avg_mempool_size.png b/cometbft/v0.38/docs/qa/img37/200nodes_cmt037/avg_mempool_size.png new file mode 100644 index 00000000..5a0f1a7b Binary files /dev/null and b/cometbft/v0.38/docs/qa/img37/200nodes_cmt037/avg_mempool_size.png differ diff --git a/cometbft/v0.38/docs/qa/img37/200nodes_cmt037/block_rate.png b/cometbft/v0.38/docs/qa/img37/200nodes_cmt037/block_rate.png new file mode 100644 index 00000000..13ddf74b Binary files /dev/null and b/cometbft/v0.38/docs/qa/img37/200nodes_cmt037/block_rate.png differ diff --git a/cometbft/v0.38/docs/qa/img37/200nodes_cmt037/cpu.png b/cometbft/v0.38/docs/qa/img37/200nodes_cmt037/cpu.png new file mode 100644 index 00000000..518bc96b Binary files /dev/null and b/cometbft/v0.38/docs/qa/img37/200nodes_cmt037/cpu.png differ diff --git a/cometbft/v0.38/docs/qa/img37/200nodes_cmt037/e_75cb89a8-f876-4698-82f3-8aaab0b361af.png b/cometbft/v0.38/docs/qa/img37/200nodes_cmt037/e_75cb89a8-f876-4698-82f3-8aaab0b361af.png new file mode 100644 index 00000000..0909fb0e Binary files /dev/null and b/cometbft/v0.38/docs/qa/img37/200nodes_cmt037/e_75cb89a8-f876-4698-82f3-8aaab0b361af.png differ diff --git a/cometbft/v0.38/docs/qa/img37/200nodes_cmt037/memory.png b/cometbft/v0.38/docs/qa/img37/200nodes_cmt037/memory.png new file mode 100644 index 00000000..c31affad Binary files /dev/null and b/cometbft/v0.38/docs/qa/img37/200nodes_cmt037/memory.png differ diff --git a/cometbft/v0.38/docs/qa/img37/200nodes_cmt037/mempool_size.png b/cometbft/v0.38/docs/qa/img37/200nodes_cmt037/mempool_size.png new file mode 100644 index 00000000..92f620df Binary files /dev/null and b/cometbft/v0.38/docs/qa/img37/200nodes_cmt037/mempool_size.png differ diff --git a/cometbft/v0.38/docs/qa/img37/200nodes_cmt037/peers.png b/cometbft/v0.38/docs/qa/img37/200nodes_cmt037/peers.png new file mode 100644 index 00000000..61d3e76c Binary files /dev/null and b/cometbft/v0.38/docs/qa/img37/200nodes_cmt037/peers.png differ diff --git a/cometbft/v0.38/docs/qa/img37/200nodes_cmt037/rounds.png b/cometbft/v0.38/docs/qa/img37/200nodes_cmt037/rounds.png new file mode 100644 index 00000000..f6bc5144 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img37/200nodes_cmt037/rounds.png differ diff --git a/cometbft/v0.38/docs/qa/img37/200nodes_cmt037/total_txs_rate.png b/cometbft/v0.38/docs/qa/img37/200nodes_cmt037/total_txs_rate.png new file mode 100644 index 00000000..d3c0dbef Binary files /dev/null and b/cometbft/v0.38/docs/qa/img37/200nodes_cmt037/total_txs_rate.png differ diff --git a/cometbft/v0.38/docs/qa/img37/200nodes_tm037/avg_mempool_size.png b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/avg_mempool_size.png new file mode 100644 index 00000000..9d95b25d Binary files /dev/null and b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/avg_mempool_size.png differ diff --git a/cometbft/v0.38/docs/qa/img37/200nodes_tm037/block_rate_regular.png b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/block_rate_regular.png new file mode 100644 index 00000000..999288e3 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/block_rate_regular.png differ diff --git a/cometbft/v0.38/docs/qa/img37/200nodes_tm037/cpu.png b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/cpu.png new file mode 100644 index 00000000..6a11aeed Binary files /dev/null and b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/cpu.png differ diff --git a/cometbft/v0.38/docs/qa/img37/200nodes_tm037/memory.png b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/memory.png new file mode 100644 index 00000000..98f5893d Binary files /dev/null and b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/memory.png differ diff --git a/cometbft/v0.38/docs/qa/img37/200nodes_tm037/mempool_size.png b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/mempool_size.png new file mode 100644 index 00000000..7b0d9e29 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/mempool_size.png differ diff --git a/cometbft/v0.38/docs/qa/img37/200nodes_tm037/peers.png b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/peers.png new file mode 100644 index 00000000..ff566edf Binary files /dev/null and b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/peers.png differ diff --git a/cometbft/v0.38/docs/qa/img37/200nodes_tm037/rounds.png b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/rounds.png new file mode 100644 index 00000000..deb201f7 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/rounds.png differ diff --git a/cometbft/v0.38/docs/qa/img37/200nodes_tm037/total_txs_rate_regular.png b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/total_txs_rate_regular.png new file mode 100644 index 00000000..08316dad Binary files /dev/null and b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/total_txs_rate_regular.png differ diff --git a/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_200node_latencies.png b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_200node_latencies.png new file mode 100644 index 00000000..ad469bb2 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_200node_latencies.png differ diff --git a/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_latency_throughput.png b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_latency_throughput.png new file mode 100644 index 00000000..baf34b2c Binary files /dev/null and b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_latency_throughput.png differ diff --git a/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_r200c2_heights.png b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_r200c2_heights.png new file mode 100644 index 00000000..360283f1 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_r200c2_heights.png differ diff --git a/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_r200c2_load1.png b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_r200c2_load1.png new file mode 100644 index 00000000..11d6dfcf Binary files /dev/null and b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_r200c2_load1.png differ diff --git a/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_r200c2_mempool_size.png b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_r200c2_mempool_size.png new file mode 100644 index 00000000..a2f3bd40 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_r200c2_mempool_size.png differ diff --git a/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_r200c2_mempool_size_avg.png b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_r200c2_mempool_size_avg.png new file mode 100644 index 00000000..480d4aeb Binary files /dev/null and b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_r200c2_mempool_size_avg.png differ diff --git a/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_r200c2_peers.png b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_r200c2_peers.png new file mode 100644 index 00000000..222da73f Binary files /dev/null and b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_r200c2_peers.png differ diff --git a/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_r200c2_rounds.png b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_r200c2_rounds.png new file mode 100644 index 00000000..7afaaac5 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_r200c2_rounds.png differ diff --git a/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_r200c2_rss.png b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_r200c2_rss.png new file mode 100644 index 00000000..730a1bc4 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_r200c2_rss.png differ diff --git a/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_r200c2_rss_avg.png b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_r200c2_rss_avg.png new file mode 100644 index 00000000..3f6cf9f6 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_r200c2_rss_avg.png differ diff --git a/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_r200c2_total-txs.png b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_r200c2_total-txs.png new file mode 100644 index 00000000..62dced2c Binary files /dev/null and b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_r200c2_total-txs.png differ diff --git a/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_report_tabbed.txt b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_report_tabbed.txt new file mode 100644 index 00000000..aa4aa4e6 --- /dev/null +++ b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_report_tabbed.txt @@ -0,0 +1,52 @@ +Experiment ID: af129eae-7039-4c76-8c37-cff9ac636a84 │Experiment ID: 0f88bd33-9bf0-4197-8d1d-9a737c301ec6 │Experiment ID: 88227cad-2ba8-4eb6-b493-041d8120b46f + │ │ + Connections: 1 │ Connections: 2 │ Connections: 4 + Rate: 25 │ Rate: 25 │ Rate: 25 + Size: 1024 │ Size: 1024 │ Size: 1024 + │ │ + Total Valid Tx: 2225 │ Total Valid Tx: 4450 │ Total Valid Tx: 8900 + Total Negative Latencies: 0 │ Total Negative Latencies: 0 │ Total Negative Latencies: 0 + Minimum Latency: 506.248587ms │ Minimum Latency: 469.53452ms │ Minimum Latency: 588.900721ms + Maximum Latency: 3.032125789s │ Maximum Latency: 6.548830955s │ Maximum Latency: 6.533739843s + Average Latency: 1.427767726s │ Average Latency: 1.448582257s │ Average Latency: 1.717432341s + Standard Deviation: 524.11782ms │ Standard Deviation: 768.684133ms │ Standard Deviation: 1.000015768s + │ │ +Experiment ID: f03d39bd-0233-4b3c-b461-543445ae1d4b │Experiment ID: 46674f1c-e591-4e36-bb9b-f375c19fc475 │Experiment ID: 5385c159-8d4d-455b-bced-dcd4a3209988 + │ │ + Connections: 1 │ Connections: 2 │ Connections: 4 + Rate: 50 │ Rate: 50 │ Rate: 50 + Size: 1024 │ Size: 1024 │ Size: 1024 + │ │ + Total Valid Tx: 4450 │ Total Valid Tx: 8900 │ Total Valid Tx: 17800 + Total Negative Latencies: 0 │ Total Negative Latencies: 0 │ Total Negative Latencies: 0 + Minimum Latency: 477.46027ms │ Minimum Latency: 455.757111ms │ Minimum Latency: 594.749081ms + Maximum Latency: 2.483895394s │ Maximum Latency: 2.904715695s │ Maximum Latency: 9.294950389s + Average Latency: 1.407374662s │ Average Latency: 1.397385779s │ Average Latency: 2.621122536s + Standard Deviation: 505.150067ms │ Standard Deviation: 551.67603ms │ Standard Deviation: 1.772725794s + │ │ +Experiment ID: 9161b4a7-d75c-455f-b82d-2b5235d533cf │Experiment ID: 993a13a8-9db1-4b2b-9c20-71a5b85e4bbf │Experiment ID: ad1eb9e1-f4d6-41fd-9ba7-0f1f7dde1e3e + │ │ + Connections: 1 │ Connections: 2 │ Connections: 4 + Rate: 100 │ Rate: 100 │ Rate: 100 + Size: 1024 │ Size: 1024 │ Size: 1024 + │ │ + Total Valid Tx: 8900 │ Total Valid Tx: 17800 │ Total Valid Tx: 35400 + Total Negative Latencies: 0 │ Total Negative Latencies: 0 │ Total Negative Latencies: 0 + Minimum Latency: 448.050467ms │ Minimum Latency: 605.436195ms │ Minimum Latency: 1.16816912s + Maximum Latency: 3.789711139s │ Maximum Latency: 7.292770222s │ Maximum Latency: 11.378681842s + Average Latency: 1.451342158s │ Average Latency: 2.07457999s │ Average Latency: 3.918384209s + Standard Deviation: 644.075973ms │ Standard Deviation: 1.230204022s │ Standard Deviation: 2.172400458s + │ │ +Experiment ID: 3cbe9c3d-9c43-4c9f-b5ca-b567d20bbd57 │Experiment ID: af836c5e-d9b6-4d5d-971c-2fc7f07aa2a0 │Experiment ID: 77606397-4989-41d4-b13b-f1f4d1af063f + │ │ + Connections: 1 │ Connections: 2 │ Connections: 4 + Rate: 200 │ Rate: 200 │ Rate: 200 + Size: 1024 │ Size: 1024 │ Size: 1024 + │ │ + Total Valid Tx: 17800 │ Total Valid Tx: 35600 │ Total Valid Tx: 37358 + Total Negative Latencies: 0 │ Total Negative Latencies: 0 │ Total Negative Latencies: 0 + Minimum Latency: 519.984701ms │ Minimum Latency: 820.755087ms │ Minimum Latency: 1.712574804s + Maximum Latency: 12.609056712s │ Maximum Latency: 9.260798095s │ Maximum Latency: 25.739223696s + Average Latency: 2.717853101s │ Average Latency: 3.477731881s │ Average Latency: 8.547725264s + Standard Deviation: 2.390778155s │ Standard Deviation: 1.675000913s │ Standard Deviation: 4.76961569s + diff --git a/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_rotating_heights.png b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_rotating_heights.png new file mode 100644 index 00000000..882de51e Binary files /dev/null and b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_rotating_heights.png differ diff --git a/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_rotating_heights_ephe.png b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_rotating_heights_ephe.png new file mode 100644 index 00000000..1ab2521e Binary files /dev/null and b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_rotating_heights_ephe.png differ diff --git a/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_rotating_latencies.png b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_rotating_latencies.png new file mode 100644 index 00000000..94548c8b Binary files /dev/null and b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_rotating_latencies.png differ diff --git a/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_rotating_load1.png b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_rotating_load1.png new file mode 100644 index 00000000..03b7412d Binary files /dev/null and b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_rotating_load1.png differ diff --git a/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_rotating_peers.png b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_rotating_peers.png new file mode 100644 index 00000000..86304760 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_rotating_peers.png differ diff --git a/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_rotating_rss_avg.png b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_rotating_rss_avg.png new file mode 100644 index 00000000..d45c045b Binary files /dev/null and b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_rotating_rss_avg.png differ diff --git a/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_rotating_total-txs.png b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_rotating_total-txs.png new file mode 100644 index 00000000..50b4c2e3 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img37/200nodes_tm037/v037_rotating_total-txs.png differ diff --git a/cometbft/v0.38/docs/qa/img37/rotating/rotating_avg_memory.png b/cometbft/v0.38/docs/qa/img37/rotating/rotating_avg_memory.png new file mode 100644 index 00000000..7feb2e81 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img37/rotating/rotating_avg_memory.png differ diff --git a/cometbft/v0.38/docs/qa/img37/rotating/rotating_block_rate.png b/cometbft/v0.38/docs/qa/img37/rotating/rotating_block_rate.png new file mode 100644 index 00000000..4bbc3c99 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img37/rotating/rotating_block_rate.png differ diff --git a/cometbft/v0.38/docs/qa/img37/rotating/rotating_cpu.png b/cometbft/v0.38/docs/qa/img37/rotating/rotating_cpu.png new file mode 100644 index 00000000..ef4c7d30 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img37/rotating/rotating_cpu.png differ diff --git a/cometbft/v0.38/docs/qa/img37/rotating/rotating_eph_heights.png b/cometbft/v0.38/docs/qa/img37/rotating/rotating_eph_heights.png new file mode 100644 index 00000000..36850fb5 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img37/rotating/rotating_eph_heights.png differ diff --git a/cometbft/v0.38/docs/qa/img37/rotating/rotating_peers.png b/cometbft/v0.38/docs/qa/img37/rotating/rotating_peers.png new file mode 100644 index 00000000..c45a4d4d Binary files /dev/null and b/cometbft/v0.38/docs/qa/img37/rotating/rotating_peers.png differ diff --git a/cometbft/v0.38/docs/qa/img37/rotating/rotating_txs_rate.png b/cometbft/v0.38/docs/qa/img37/rotating/rotating_txs_rate.png new file mode 100644 index 00000000..5462738e Binary files /dev/null and b/cometbft/v0.38/docs/qa/img37/rotating/rotating_txs_rate.png differ diff --git a/cometbft/v0.38/docs/qa/img38/200nodes/avg_mempool_size.png b/cometbft/v0.38/docs/qa/img38/200nodes/avg_mempool_size.png new file mode 100644 index 00000000..36cd5f6c Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/200nodes/avg_mempool_size.png differ diff --git a/cometbft/v0.38/docs/qa/img38/200nodes/block_rate.png b/cometbft/v0.38/docs/qa/img38/200nodes/block_rate.png new file mode 100644 index 00000000..b2042865 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/200nodes/block_rate.png differ diff --git a/cometbft/v0.38/docs/qa/img38/200nodes/c1r400.png b/cometbft/v0.38/docs/qa/img38/200nodes/c1r400.png new file mode 100644 index 00000000..0c27c144 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/200nodes/c1r400.png differ diff --git a/cometbft/v0.38/docs/qa/img38/200nodes/cpu.png b/cometbft/v0.38/docs/qa/img38/200nodes/cpu.png new file mode 100644 index 00000000..15f74aeb Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/200nodes/cpu.png differ diff --git a/cometbft/v0.38/docs/qa/img38/200nodes/e_de676ecf-038e-443f-a26a-27915f29e312.png b/cometbft/v0.38/docs/qa/img38/200nodes/e_de676ecf-038e-443f-a26a-27915f29e312.png new file mode 100644 index 00000000..21f5cab6 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/200nodes/e_de676ecf-038e-443f-a26a-27915f29e312.png differ diff --git a/cometbft/v0.38/docs/qa/img38/200nodes/memory.png b/cometbft/v0.38/docs/qa/img38/200nodes/memory.png new file mode 100644 index 00000000..3e01c4cc Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/200nodes/memory.png differ diff --git a/cometbft/v0.38/docs/qa/img38/200nodes/mempool_size.png b/cometbft/v0.38/docs/qa/img38/200nodes/mempool_size.png new file mode 100644 index 00000000..6a897a84 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/200nodes/mempool_size.png differ diff --git a/cometbft/v0.38/docs/qa/img38/200nodes/peers.png b/cometbft/v0.38/docs/qa/img38/200nodes/peers.png new file mode 100644 index 00000000..87470051 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/200nodes/peers.png differ diff --git a/cometbft/v0.38/docs/qa/img38/200nodes/rounds.png b/cometbft/v0.38/docs/qa/img38/200nodes/rounds.png new file mode 100644 index 00000000..5ac53cad Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/200nodes/rounds.png differ diff --git a/cometbft/v0.38/docs/qa/img38/200nodes/total_txs_rate.png b/cometbft/v0.38/docs/qa/img38/200nodes/total_txs_rate.png new file mode 100644 index 00000000..ac7f5e68 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/200nodes/total_txs_rate.png differ diff --git a/cometbft/v0.38/docs/qa/img38/200nodes/v038_report_tabbed.txt b/cometbft/v0.38/docs/qa/img38/200nodes/v038_report_tabbed.txt new file mode 100644 index 00000000..c482aeac --- /dev/null +++ b/cometbft/v0.38/docs/qa/img38/200nodes/v038_report_tabbed.txt @@ -0,0 +1,40 @@ +Experiment ID: 93024f38-a008-443d-9aa7-9ac44c9fe15b Experiment ID: d65a486e-4712-41b5-9f41-97e491895d2e Experiment ID: 9c39184b-b8c7-46a2-bacb-40f9961fb7a1 + Connections: 1 Connections: 2 Connections: 4 + Rate: 200 Rate: 200 Rate: 200 + Size: 1024 Size: 1024 Size: 1024 + Total Valid Tx: 17800 Total Valid Tx: 33259 Total Valid Tx: 33259 + Total Negative Latencies: 0 Total Negative Latencies: 0 Total Negative Latencies: 0 + Minimum Latency: 562.805076ms Minimum Latency: 894.026089ms Minimum Latency: 2.166875257s + Maximum Latency: 7.623963559s Maximum Latency: 16.941216187s Maximum Latency: 15.701598288s + Average Latency: 1.860012628s Average Latency: 4.033134276s Average Latency: 7.592412668s + Standard Deviation: 1.169158915s Standard Deviation: 3.427243686s Standard Deviation: 2.951797195s +Experiment ID: de676ecf-038e-443f-a26a-27915f29e312 Experiment ID: 39d571b8-f39b-4aec-bd6a-e94f28a42a63 Experiment ID: 5b855105-60b5-4c2d-ba5c-fdad0213765c + Connections: 1 Connections: 2 Connections: 4 + Rate: 400 Rate: 400 Rate: 400 + Size: 1024 Size: 1024 Size: 1024 + Total Valid Tx: 35600 Total Valid Tx: 41565 Total Valid Tx: 41384 + Total Negative Latencies: 0 Total Negative Latencies: 0 Total Negative Latencies: 0 + Minimum Latency: 565.640641ms Minimum Latency: 1.650712046s Minimum Latency: 2.796290248s + Maximum Latency: 10.051316705s Maximum Latency: 15.897581951s Maximum Latency: 20.124431723s + Average Latency: 3.499369173s Average Latency: 8.635543807s Average Latency: 10.596146863s + Standard Deviation: 1.926805844s Standard Deviation: 2.535678364s Standard Deviation: 3.193742233s +Experiment ID: db10ca9e-6cf8-4dc9-9284-6e767e4b4346 Experiment ID: f57af87d-d342-41f7-a0eb-baa87a4b2257 Experiment ID: 32819ea0-1a59-41de-8aa6-b70f68697520 + Connections: 1 Connections: 2 Connections: 4 + Rate: 800 Rate: 800 Rate: 800 + Size: 1024 Size: 1024 Size: 1024 + Total Valid Tx: 36831 Total Valid Tx: 38686 Total Valid Tx: 40816 + Total Negative Latencies: 0 Total Negative Latencies: 0 Total Negative Latencies: 0 + Minimum Latency: 1.203966853s Minimum Latency: 728.863446ms Minimum Latency: 1.559342549s + Maximum Latency: 21.411365818s Maximum Latency: 24.349050642s Maximum Latency: 25.791215028s + Average Latency: 9.213156739s Average Latency: 11.194994374s Average Latency: 11.950851892s + Standard Deviation: 4.909584729s Standard Deviation: 5.199186587s Standard Deviation: 4.315394253s +Experiment ID: 587762c4-3fd4-4799-9f3b-9e6971b353ba Experiment ID: 489b2623-a3e4-453f-a771-5d05e7de4a1f Experiment ID: 98605df2-3b16-46db-8675-2980bc84ea2b + Connections: 1 Connections: 2 Connections: 4 + Rate: 1600 Rate: 1600 Rate: 1600 + Size: 1024 Size: 1024 Size: 1024 + Total Valid Tx: 40600 Total Valid Tx: 45034 Total Valid Tx: 39830 + Total Negative Latencies: 0 Total Negative Latencies: 0 Total Negative Latencies: 0 + Minimum Latency: 998.07523ms Minimum Latency: 1.43819209s Minimum Latency: 1.50664776s + Maximum Latency: 18.565312759s Maximum Latency: 17.098811297s Maximum Latency: 20.346885373s + Average Latency: 8.78128586s Average Latency: 8.957419021s Average Latency: 12.113245591s + Standard Deviation: 3.305897473s Standard Deviation: 2.734640455s Standard Deviation: 4.029854219s diff --git a/cometbft/v0.38/docs/qa/img38/rotating/rotating_avg_memory.png b/cometbft/v0.38/docs/qa/img38/rotating/rotating_avg_memory.png new file mode 100644 index 00000000..43dadcab Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/rotating/rotating_avg_memory.png differ diff --git a/cometbft/v0.38/docs/qa/img38/rotating/rotating_block_rate.png b/cometbft/v0.38/docs/qa/img38/rotating/rotating_block_rate.png new file mode 100644 index 00000000..e627064a Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/rotating/rotating_block_rate.png differ diff --git a/cometbft/v0.38/docs/qa/img38/rotating/rotating_cpu.png b/cometbft/v0.38/docs/qa/img38/rotating/rotating_cpu.png new file mode 100644 index 00000000..f51403d4 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/rotating/rotating_cpu.png differ diff --git a/cometbft/v0.38/docs/qa/img38/rotating/rotating_eph_heights.png b/cometbft/v0.38/docs/qa/img38/rotating/rotating_eph_heights.png new file mode 100644 index 00000000..6c8e08ee Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/rotating/rotating_eph_heights.png differ diff --git a/cometbft/v0.38/docs/qa/img38/rotating/rotating_latencies.png b/cometbft/v0.38/docs/qa/img38/rotating/rotating_latencies.png new file mode 100644 index 00000000..8031d7cd Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/rotating/rotating_latencies.png differ diff --git a/cometbft/v0.38/docs/qa/img38/rotating/rotating_peers.png b/cometbft/v0.38/docs/qa/img38/rotating/rotating_peers.png new file mode 100644 index 00000000..b0ac6a2b Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/rotating/rotating_peers.png differ diff --git a/cometbft/v0.38/docs/qa/img38/rotating/rotating_txs_rate.png b/cometbft/v0.38/docs/qa/img38/rotating/rotating_txs_rate.png new file mode 100644 index 00000000..d3ae7141 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/rotating/rotating_txs_rate.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/02k_1_block_rate.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/02k_1_block_rate.png new file mode 100644 index 00000000..ff314bd7 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/02k_1_block_rate.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/02k_1_total_txs_rate.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/02k_1_total_txs_rate.png new file mode 100644 index 00000000..67a0cbb5 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/02k_1_total_txs_rate.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/02k_avg_cpu.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/02k_avg_cpu.png new file mode 100644 index 00000000..2eb9683e Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/02k_avg_cpu.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/02k_avg_memory.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/02k_avg_memory.png new file mode 100644 index 00000000..955d1136 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/02k_avg_memory.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/02k_avg_mempool_size.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/02k_avg_mempool_size.png new file mode 100644 index 00000000..426867db Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/02k_avg_mempool_size.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/02k_block_rate.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/02k_block_rate.png new file mode 100644 index 00000000..0a53ccf7 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/02k_block_rate.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/02k_rounds.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/02k_rounds.png new file mode 100644 index 00000000..b932e80d Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/02k_rounds.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/02k_total_txs_rate.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/02k_total_txs_rate.png new file mode 100644 index 00000000..c7040fb8 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/02k_total_txs_rate.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/04k_1_block_rate.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/04k_1_block_rate.png new file mode 100644 index 00000000..9f9564a6 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/04k_1_block_rate.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/04k_1_total_txs_rate.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/04k_1_total_txs_rate.png new file mode 100644 index 00000000..e69096d7 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/04k_1_total_txs_rate.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/04k_avg_cpu.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/04k_avg_cpu.png new file mode 100644 index 00000000..be251929 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/04k_avg_cpu.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/04k_avg_memory.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/04k_avg_memory.png new file mode 100644 index 00000000..50503a3a Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/04k_avg_memory.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/04k_avg_mempool_size.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/04k_avg_mempool_size.png new file mode 100644 index 00000000..3e3eea8e Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/04k_avg_mempool_size.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/04k_block_rate.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/04k_block_rate.png new file mode 100644 index 00000000..f0bd5c2a Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/04k_block_rate.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/04k_rounds.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/04k_rounds.png new file mode 100644 index 00000000..64bb878f Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/04k_rounds.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/04k_total_txs_rate.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/04k_total_txs_rate.png new file mode 100644 index 00000000..be5ab70a Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/04k_total_txs_rate.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/08k_1_avg_mempool_size.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/08k_1_avg_mempool_size.png new file mode 100644 index 00000000..00cc236c Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/08k_1_avg_mempool_size.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/08k_1_block_rate.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/08k_1_block_rate.png new file mode 100644 index 00000000..9caa120f Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/08k_1_block_rate.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/08k_1_rounds.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/08k_1_rounds.png new file mode 100644 index 00000000..809a97ee Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/08k_1_rounds.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/08k_1_total_txs_rate.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/08k_1_total_txs_rate.png new file mode 100644 index 00000000..ce1c5c7d Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/08k_1_total_txs_rate.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/08k_avg_cpu.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/08k_avg_cpu.png new file mode 100644 index 00000000..c78af4f2 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/08k_avg_cpu.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/08k_avg_memory.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/08k_avg_memory.png new file mode 100644 index 00000000..cd36d056 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/08k_avg_memory.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/08k_avg_mempool_size.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/08k_avg_mempool_size.png new file mode 100644 index 00000000..dd852e9b Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/08k_avg_mempool_size.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/08k_rounds.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/08k_rounds.png new file mode 100644 index 00000000..0bd98303 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/08k_rounds.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/08k_total_txs_rate.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/08k_total_txs_rate.png new file mode 100644 index 00000000..87cb6e4b Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/08k_total_txs_rate.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/16k_1_avg_mempool_size.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/16k_1_avg_mempool_size.png new file mode 100644 index 00000000..3eb5b73d Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/16k_1_avg_mempool_size.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/16k_1_block_rate.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/16k_1_block_rate.png new file mode 100644 index 00000000..12025af8 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/16k_1_block_rate.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/16k_1_rounds.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/16k_1_rounds.png new file mode 100644 index 00000000..15d6feb2 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/16k_1_rounds.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/16k_1_total_txs_rate.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/16k_1_total_txs_rate.png new file mode 100644 index 00000000..65cb0c11 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/16k_1_total_txs_rate.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/16k_avg_cpu.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/16k_avg_cpu.png new file mode 100644 index 00000000..6fff44da Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/16k_avg_cpu.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/16k_avg_memory.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/16k_avg_memory.png new file mode 100644 index 00000000..218ef0bd Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/16k_avg_memory.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/16k_avg_mempool_size.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/16k_avg_mempool_size.png new file mode 100644 index 00000000..73881a15 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/16k_avg_mempool_size.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/16k_block_rate.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/16k_block_rate.png new file mode 100644 index 00000000..73cbba28 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/16k_block_rate.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/16k_rounds.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/16k_rounds.png new file mode 100644 index 00000000..7458188b Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/16k_rounds.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/16k_total_txs_rate.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/16k_total_txs_rate.png new file mode 100644 index 00000000..5d44a422 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/16k_total_txs_rate.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/32k_1_avg_mempool_size.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/32k_1_avg_mempool_size.png new file mode 100644 index 00000000..273f1b8b Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/32k_1_avg_mempool_size.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/32k_1_block_rate.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/32k_1_block_rate.png new file mode 100644 index 00000000..d469e947 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/32k_1_block_rate.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/32k_1_rounds.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/32k_1_rounds.png new file mode 100644 index 00000000..347263dd Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/32k_1_rounds.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/32k_1_total_txs_rate.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/32k_1_total_txs_rate.png new file mode 100644 index 00000000..0d0451ac Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/32k_1_total_txs_rate.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/32k_avg_cpu.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/32k_avg_cpu.png new file mode 100644 index 00000000..5464681c Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/32k_avg_cpu.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/32k_avg_memory.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/32k_avg_memory.png new file mode 100644 index 00000000..4cea5df7 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/32k_avg_memory.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/32k_avg_mempool_size.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/32k_avg_mempool_size.png new file mode 100644 index 00000000..e573eca2 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/32k_avg_mempool_size.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/32k_block_rate.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/32k_block_rate.png new file mode 100644 index 00000000..f3ebf625 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/32k_block_rate.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/32k_rounds.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/32k_rounds.png new file mode 100644 index 00000000..a7a0597c Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/32k_rounds.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/32k_total_txs_rate.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/32k_total_txs_rate.png new file mode 100644 index 00000000..fdd93e25 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/32k_total_txs_rate.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/8k_block_rate.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/8k_block_rate.png new file mode 100644 index 00000000..da0a378a Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/8k_block_rate.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/all_c1r400_16k.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/all_c1r400_16k.png new file mode 100644 index 00000000..d7e18134 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/all_c1r400_16k.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/all_c1r400_2k.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/all_c1r400_2k.png new file mode 100644 index 00000000..9682d84a Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/all_c1r400_2k.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/all_c1r400_32k.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/all_c1r400_32k.png new file mode 100644 index 00000000..5179fd21 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/all_c1r400_32k.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/all_c1r400_4k.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/all_c1r400_4k.png new file mode 100644 index 00000000..46d67719 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/all_c1r400_4k.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/all_c1r400_64k.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/all_c1r400_64k.png new file mode 100644 index 00000000..04ff7e76 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/all_c1r400_64k.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/all_c1r400_8k.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/all_c1r400_8k.png new file mode 100644 index 00000000..b54ed7d8 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/all_c1r400_8k.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/all_c1r400_baseline.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/all_c1r400_baseline.png new file mode 100644 index 00000000..2c094f58 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/all_c1r400_baseline.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/all_experiments_16k.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/all_experiments_16k.png new file mode 100644 index 00000000..c7a1f6da Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/all_experiments_16k.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/all_experiments_2k.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/all_experiments_2k.png new file mode 100644 index 00000000..ea717d55 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/all_experiments_2k.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/all_experiments_32k.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/all_experiments_32k.png new file mode 100644 index 00000000..ee4cf339 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/all_experiments_32k.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/all_experiments_4k.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/all_experiments_4k.png new file mode 100644 index 00000000..eb4c569c Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/all_experiments_4k.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/all_experiments_64k.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/all_experiments_64k.png new file mode 100644 index 00000000..f7abbae5 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/all_experiments_64k.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/all_experiments_8k.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/all_experiments_8k.png new file mode 100644 index 00000000..cbaaf5c9 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/all_experiments_8k.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/all_experiments_baseline.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/all_experiments_baseline.png new file mode 100644 index 00000000..b27ec5d6 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/all_experiments_baseline.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/baseline_1_avg_mempool_size.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/baseline_1_avg_mempool_size.png new file mode 100644 index 00000000..63c86687 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/baseline_1_avg_mempool_size.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/baseline_1_block_rate.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/baseline_1_block_rate.png new file mode 100644 index 00000000..46f0a4ee Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/baseline_1_block_rate.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/baseline_1_rounds.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/baseline_1_rounds.png new file mode 100644 index 00000000..1e6db5e3 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/baseline_1_rounds.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/baseline_1_total_txs_rate.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/baseline_1_total_txs_rate.png new file mode 100644 index 00000000..75f9ab43 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/baseline_1_total_txs_rate.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/baseline_avg_cpu.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/baseline_avg_cpu.png new file mode 100644 index 00000000..2c1bca8b Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/baseline_avg_cpu.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/baseline_avg_memory.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/baseline_avg_memory.png new file mode 100644 index 00000000..f0529880 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/baseline_avg_memory.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/baseline_avg_mempool_size.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/baseline_avg_mempool_size.png new file mode 100644 index 00000000..179693cc Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/baseline_avg_mempool_size.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/baseline_block_rate.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/baseline_block_rate.png new file mode 100644 index 00000000..20073522 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/baseline_block_rate.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/baseline_rounds.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/baseline_rounds.png new file mode 100644 index 00000000..468d4e2f Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/baseline_rounds.png differ diff --git a/cometbft/v0.38/docs/qa/img38/voteExtensions/baseline_total_txs_rate.png b/cometbft/v0.38/docs/qa/img38/voteExtensions/baseline_total_txs_rate.png new file mode 100644 index 00000000..306793d5 Binary files /dev/null and b/cometbft/v0.38/docs/qa/img38/voteExtensions/baseline_total_txs_rate.png differ diff --git a/cometbft/v0.38/docs/rfc/README.md b/cometbft/v0.38/docs/rfc/README.md new file mode 100644 index 00000000..6693c1ff --- /dev/null +++ b/cometbft/v0.38/docs/rfc/README.md @@ -0,0 +1,45 @@ +--- +order: 1 +parent: + order: false +--- + +# Requests for Comments + +A Request for Comments (RFC) is a record of discussion on an open-ended topic +related to the design and implementation of CometBFT, for which no +immediate decision is required. + +The purpose of an RFC is to serve as a historical record of a high-level +discussion that might otherwise only be recorded in an ad-hoc way (for example, +via gists or Google docs) that are difficult to discover for someone after the +fact. An RFC _may_ give rise to more specific architectural _decisions_ for +CometBFT, but those decisions must be recorded separately in +[Architecture Decision Records (ADR)](../architecture/). + +As a rule of thumb, if you can articulate a specific question that needs to be +answered, write an ADR. If you need to explore the topic and get input from +others to know what questions need to be answered, an RFC may be appropriate. + +## RFC Content + +An RFC should provide: + +- A **changelog**, documenting when and how the RFC has changed. +- An **abstract**, briefly summarizing the topic so the reader can quickly tell + whether it is relevant to their interest. +- Any **background** a reader will need to understand and participate in the + substance of the discussion (links to other documents are fine here). +- The **discussion**, the primary content of the document. + +The [rfc-template.md](./rfc-template.md) file includes placeholders for these +sections. + +## Table of Contents + +The RFCs listed below are exclusively relevant to CometBFT. For historical RFCs +relating to Tendermint Core prior to forking, please see +[this list](./tendermint-core/). + + +- [RFC-100: ABCI Vote Extension Propagation](./rfc-100-abci-vote-extension-propag.md) diff --git a/cometbft/v0.38/docs/rfc/rfc-100-abci-vote-extension-propag.md b/cometbft/v0.38/docs/rfc/rfc-100-abci-vote-extension-propag.md new file mode 100644 index 00000000..fce0fe9a --- /dev/null +++ b/cometbft/v0.38/docs/rfc/rfc-100-abci-vote-extension-propag.md @@ -0,0 +1,742 @@ +# RFC 100: ABCI Vote Extension Propagation + +## Changelog + +- 11-Apr-2022: Initial draft (@sergio-mena). +- 15-Apr-2022: Addressed initial comments. First complete version (@sergio-mena). +- 09-May-2022: Addressed all outstanding comments (@sergio-mena). +- 09-May-2022: Add section on upgrade path (@wbanfield) +- 02-Mar-2023: Migrated to CometBFT RFCs. New number: RFC 100 (@sergio-mena). +- 03-Mar-2023: Added "changes needed" to solutions in upgrade path section (@sergio-mena) + +## Abstract + +According to the +[ABCI 2.0 specification][abci-2-0], +a validator MUST provide a signed vote extension for each non-`nil` precommit vote +of height *h* that it uses to propose a block in height *h+1*. When a validator is up to +date, this is easy to do, but when a validator needs to catch up this is far from trivial as this data +cannot be retrieved from the blockchain. + +This RFC presents and compares the different options to address this problem, which have been proposed +in several discussions by the CometBFT team. + +## Document Structure + +The RFC is structured as follows. In the [Background](#background) section, +subsections [Problem Description](#problem-description) and [Cases to Address](#cases-to-address) +explain the problem at hand from a high level perspective, i.e., abstracting away from the current +CometBFT implementation. In contrast, subsection +[Current Catch-up Mechanisms](#current-catch-up-mechanisms) delves into the details of the current +CometBFT code. + +In the [Discussion](#discussion) section, subsection [Solutions Proposed](#solutions-proposed) is also +worded abstracting away from implementation details, whilst subsections +[Feasibility of the Proposed Solutions](#feasibility-of-the-proposed-solutions) and +[Current Limitations and Possible Implementations](#current-limitations-and-possible-implementations) +analyze the viability of one of the proposed solutions in the context of CometBFT's architecture +based on reactors. +Subsection [Upgrade Path](#upgrade-path) discusses how a CometBFT node can upgrade +from a version predating vote extensions, to one featuring it. +Finally, [Formalization Work](#formalization-work) briefly discusses the work +still needed to demonstrate the correctness of the chosen solution. + +The high level subsections are aimed at readers who are familiar with consensus algorithms, in +particular with the Tendermint algorithm described [here](https://arxiv.org/abs/1807.04938), +but who are not necessarily +acquainted with the details of the CometBFT codebase. The other subsections, which go into +implementation details, are best understood by engineers with deep knowledge of the implementation of +CometBFT's blocksync and consensus reactors. + +## Background + +### Basic Definitions + +This document assumes that all validators have equal voting power for the sake of simplicity. This is done +without loss of generality. + +There are two types of votes in the Tendermint algorithm: *prevotes* and *precommits*. +Votes can be `nil` or refer to a proposed block. This RFC focuses on precommits, +also known as *precommit votes*. In this document we sometimes call them simply *votes*. + +Validators send precommit votes to their peer nodes in *precommit messages*. According to the +[ABCI 2.0 specification][abci-2-0], +a precommit message MUST also contain a *vote extension*. +This mandatory vote extension can be empty, but MUST be signed with the same key as the precommit +vote (i.e., the sending validator's). +Nevertheless, the vote extension is signed independently from the vote, so a vote can be separated from +its extension. +The reason for vote extensions to be mandatory in precommit messages is that, otherwise, a (malicious) +node can omit a vote extension while still providing/forwarding/sending the corresponding precommit vote. + +The validator set at height *h* is denoted *valseth*. A *commit* for height *h* consists of more +than *2nh/3* precommit votes voting for a block *b*, where *nh* denotes the size of +*valseth*. A commit does not contain `nil` precommit votes, and all votes in it refer to the +same block. An *extended commit* is a *commit* where every precommit vote has its respective vote extension +attached. + +### Problem Description + +In [ABCI 1.0][abci-1-0] and previous versions (e.g. [ABCI 0.17.0][abci-0-17-0]), +for any height *h*, a validator *v* MUST have the decided block *b* and a commit for +height *h* in order to decide at height *h*. Then, *v* just needs a commit for height *h* to propose at +height *h+1*, in the rounds of *h+1* where *v* is a proposer. + +In [ABCI 2.0][abci-2-0], +the information that a validator *v* MUST have to be able to decide in *h* does not change with +respect to pre-existing ABCI: the decided block *b* and a commit for *h*. +In contrast, for proposing in *h+1*, a commit for *h* is not enough: *v* MUST now have an extended +commit. + +When a validator takes an active part in consensus at height *h*, it has all the data it needs in memory, +in its consensus state, to decide on *h* and propose in *h+1*. Things are not so easy in the cases when +*v* cannot take part in consensus because it is late (e.g., it falls behind, it crashes +and recovers, or it just starts after the others). If *v* does not take part, it cannot actively +gather precommit messages (which include vote extensions) in order to decide. +Before ABCI 2.0, this was not a problem: full nodes are supposed to persist past blocks in the block store, +so other nodes would realise that *v* is late and send it the missing decided block at height *h* and +the corresponding commit (kept in block *h+1*) so that *v* can catch up. +However, we cannot apply this catch-up technique for ABCI 2.0, as the vote extensions, which are part +of the needed *extended commit* are not part of the blockchain. + +### Cases to Address + +Before we tackle the description of the possible cases we need to address, let us describe the following +incremental improvement to the ABCI 2.0 logic. Upon decision, a full node persists (e.g., in the block +store) the extended commit that allowed the node to decide. For the moment, let us assume the node only +needs to keep its *most recent* extended commit, and MAY remove any older extended commits from persistent +storage. +This improvement is so obvious that all solutions described in the [Discussion](#discussion) section use +it as a building block. Moreover, it completely addresses by itself some of the cases described in this +subsection. + +We now describe the cases (i.e. possible *runs* of the system) that have been raised in different +discussions and need to be addressed. They are (roughly) ordered from easiest to hardest to deal with. + +- **(a)** *Happy path: all validators advance together, no crash*. + + This case is included for completeness. All validators have taken part in height *h*. + Even if some of them did not manage to send a precommit message for the decided block, they all + receive enough precommit messages to be able to decide. As vote extensions are mandatory in + precommit messages, every validator *v* trivially has all the information, namely the decided block + and the extended commit, needed to propose in height *h+1* for the rounds in which *v* is the + proposer. + + No problem to solve here. + +- **(b)** *All validators advance together, then all crash at the same height*. + + This case has been raised in some discussions, the main concern being whether the vote extensions + for the previous height would be lost across the network. With the improvement described above, + namely persisting the latest extended commit at decision time, this case is solved. + When a crashed validator recovers, it recovers the last extended commit from persistent storage + and handshakes with the Application. + If need be, it also reconstructs messages for the unfinished height + (including all precommits received) from the WAL. + Then, the validator can resume where it was at the time of the crash. Thus, as extensions are + persisted, either in the WAL (in the form of received precommit messages), or in the latest + extended commit, the only way that vote extensions needed to start the next height could be lost + forever would be if all validators crashed and never recovered (e.g. disk corruption). + Since a *correct* node MUST eventually recover, this violates the assumption of more than + *2nh/3* correct validators for every height *h*. + + No problem to solve here. + +- **(c)** *Lagging majority*. + + Let us assume the validator set does not change between *h* and *h+1*. + It is not possible by the nature of the Tendermint algorithm, which requires more + than *2nh/3* precommit votes for some round of height *h* in order to make progress. + So, only up to *nh/3* validators can lag behind. + + On the other hand, for the case where there are changes to the validator set between *h* and + *h+1* please see case (d) below, where the extreme case is discussed. + +- **(d)** *Validator set changes completely between* h *and* h+1. + + If sets *valseth* and *valseth+1* are disjoint, + more than *2nh/3* of validators in height *h* should + have actively participated in conensus in *h*. So, as of height *h*, only a minority of validators + in *h* can be lagging behind, although they could all lag behind from *h+1* on, as they are no + longer validators, only full nodes. This situation falls under the assumptions of case (h) below. + + As for validators in *valseth+1*, as they were not validators as of height *h*, they + could all be lagging behind by that time. However, by the time *h* finishes and *h+1* begins, the + chain will halt until more than *2nh+1/3* of them have caught up and started consensus + at height *h+1*. If set *valseth+1* does not change in *h+2* and subsequent + heights, only up to *nh+1/3* validators will be able to lag behind. Thus, we have + converted this case into case (h) below. + +- **(e)** *Enough validators crash to block the rest*. + + In this case, blockchain progress halts, i.e. surviving full nodes keep increasing rounds + indefinitely, until some of the crashed validators are able to recover. + Those validators that recover first will handshake with the Application and recover at the height + they crashed, which is still the same the nodes that did not crash are stuck in, so they don't need + to catch up. + Further, they had persisted the extended commit for the previous height. Nothing to solve. + + For those validators recovering later, we are in case (h) below. + +- **(f)** *Some validators crash, but not enough to block progress*. + + When the correct processes that crashed recover, they handshake with the Application and resume at + the height they were at when they crashed. As the blockchain did not stop making progress, the + recovered processes are likely to have fallen behind with respect to the progressing majority. + + At this point, the recovered processes are in case (h) below. + +- **(g)** *A new full node starts*. + + The reasoning here also applies to the case when more than one full node are starting. + When the full node starts from scratch, it has no state (its current height is 0). Ignoring + statesync for the time being, the node just needs to catch up by applying past blocks one by one + (after verifying them). + + Thus, the node is in case (h) below. + +- **(h)** *Advancing majority, lagging minority* + + In this case, some nodes are late. More precisely, at the present time, a set of full nodes, + denoted *Lhp*, are falling behind + (e.g., temporary disconnection or network partition, memory thrashing, crashes, new nodes) + an arbitrary + number of heights: + between *hs* and *hp*, where *hs < hp*, and + *hp* is the highest height + any correct full node has reached so far. + + The correct full nodes that reached *hp* were able to decide for *hp-1*. + Therefore, less than *nhp-1/3* validators of *hp-1* can be part + of *Lhp*, since enough up-to-date validators needed to actively participate + in consensus for *hp-1*. + + Since, at the present time, + no node in *Lhp* took part in any consensus between + *hs* and *hp-1*, + the reasoning above can be extended to validator set changes between *hs* and + *hp-1*. This results in the following restriction on the full nodes that can be part of *Lhp*. + + - ∀ *h*, where *hs ≤ h < hp*, + | *valseth* ∩ *Lhp* | *< nh/3* + + So, full nodes that are validators at some height h between *hs* and *hp-1* + can be in *Lhp*, but not more than 1/3 of those acting as validators in + the same height. + If this property does not hold for a particular height *h*, where + *hs ≤ h < hp*, CometBFT could not have progressed beyond *h* and + therefore no full node could have reached *hp* (a contradiction). + + These lagging nodes in *Lhp* need to catch up. They have to obtain the + information needed to make + progress from other nodes. For each height *h* between *hs* and *hp-2*, + this includes the decided block for *h*, and the + precommit votes also for *deciding h* (which can be extracted from the block at height *h+1*). + + At a given height *hc* (where possibly *hc << hp*), + a full node in *Lhp* will consider itself *caught up*, based on the + (maybe out of date) information it is getting from its peers. Then, the node needs to be ready to + propose at height *hc+1*, which requires having received the vote extensions for + *hc*. + As the vote extensions are *not* stored in the blocks, and it is difficult to have strong + guarantees on *when* a late node considers itself caught up, providing the late node with the right + vote extensions for the right height poses a problem. + +At this point, we have described and compared all cases raised in discussions leading up to this +RFC. The list above aims at being exhaustive. The analysis of each case included above makes all of +them converge into case (h). + +### Current Catch-up Mechanisms + +We now briefly describe the current catch-up mechanisms in the reactors concerned in CometBFT. + +#### Statesync + +Full nodes optionally run statesync just after starting, when they start from scratch. +If statesync succeeds, an Application snapshot is installed, and CometBFT jumps from height 0 directly +to the height the Application snapshop represents, without applying the block of any previous height. +Some light blocks are received and stored in the block store for running light-client verification of +all the skipped blocks. Light blocks are incomplete blocks, typically containing the header and the +canonical commit but, e.g., no transactions. They are stored in the block store as "signed headers". + +The statesync reactor is not really relevant for solving the problem discussed in this RFC. We will +nevertheless mention it when needed; in particular, to understand some corner cases. + +#### Blocksync + +The blocksync reactor kicks in after start up or recovery. +At startup, if statesync is enabled, blocksync starts just after statesync +and sends the following messages to its peers: + +- `StatusRequest` to query the height its peers are currently at, and +- `BlockRequest`, asking for blocks of heights the local node is missing. + +Using `BlockResponse` messages received from peers, the blocksync reactor validates each received +block using the block of the following height, saves the block in the block store, and sends the +block to the Application for execution (it effectively simulates the node *deciding* on that height). + +If blocksync has validated and applied the block for the height *previous* to the highest seen in +a `StatusResponse` message, or if no progress has been made after a timeout, the node considers +itself as caught up and switches to the consensus reactor. + +#### Consensus Reactor + +The consensus reactor runs the full Tendermint algorithm. For a validator this means it has to +propose blocks, and send/receive prevote/precommit messages, as mandated by the algorithm, +before it can decide and move on to the next height. + +If a full node that is running the consensus reactor falls behind at height *h*, when a peer node +realises this it will retrieve the canonical commit of *h+1* from the block store, and *convert* +it into a set of precommit votes and will send those to the late node. + +## Discussion + +### Solutions Proposed + +These are the solutions proposed in discussions leading up to this RFC. + +- **Solution 0.** *Vote extensions are made **best effort** in the specification*. + + This is the simplest solution, considered as a way to provide vote extensions in a simple enough + way so that it can be a first available version in ABCI 2.0. + It consists in changing the specification so as to not *require* that precommit votes used upon + `PrepareProposal` contain their corresponding vote extensions. In other words, we render vote + extensions optional. + There are strong implications stemming from such a relaxation of the original specification. + + - As a vote extension is signed *separately* from the vote it is extending, an intermediate node + can now remove (i.e., censor) vote extensions from precommit messages at will. + - Further, there is no point anymore in the spec requiring the Application to accept a vote extension + passed via `VerifyVoteExtension` to consider a precommit message valid in its entirety. Remember + this behavior of `VerifyVoteExtension` is adding a constraint to CometBFT's conditions for + liveness. + In this situation, it is better and simpler to just drop the vote extension rejected by the + Application via `VerifyVoteExtension`, but still consider the precommit vote itself valid as long + as its signature verifies. + +- **Solution 1.** *Include vote extensions in the blockchain*. + + Another obvious solution, which has somehow been considered in the past, is to include the vote + extensions and their signatures in the blockchain. + The blockchain would thus include the extended commit, rather than a regular commit, as the structure + to be canonicalized in the next block. + With this solution, the current mechanisms implemented both in the blocksync and consensus reactors + would still be correct, as all the information a node needs to catch up, and to start proposing when + it considers itself as caught-up, can now be recovered from past blocks saved in the block store. + + This solution has two main drawbacks. + + - As the block format must change, upgrading a chain requires a hard fork. Furthermore, + all existing light client implementations will stop working until they are upgraded to deal with + the new format (e.g., how certain hashes calculated and/or how certain signatures are checked). + For instance, let us consider IBC, which relies on light clients. An IBC connection between + two chains will be broken if only one chain upgrades. + - The extra information (i.e., the vote extensions) that is now kept in the blockchain is not really + needed *at every height* for a late node to catch up. + - This information is only needed to be able to *propose* at the height the validator considers + itself as caught-up. If a validator is indeed late for height *h*, it is useless (although + correct) for it to call `PrepareProposal`, or `ExtendVote`, since the block is already decided. + - Moreover, some use cases require pretty sizeable vote extensions, which would result in an + important waste of space in the blockchain. + +- **Solution 2.** *Skip* propose *step in Tendermint algorithm*. + + This solution consists in modifying the Tendermint algorithm to skip the *send proposal* step in + heights where the node does not have the required vote extensions to populate the call to + `PrepareProposal`. The main idea behind this is that it should only happen when the validator is late + and, therefore, up-to-date validators have already proposed (and decided) for that height. + A small variation of this solution is, rather than skipping the *send proposal* step, the validator + sends a special *empty* or *bottom* (⊥) proposal to signal other nodes that it is not ready to propose + at (any round of) the current height. + + The appeal of this solution is its simplicity. A possible implementation does not need to extend + the data structures, or change the current catch-up mechanisms implemented in the blocksync or + in the consensus reactors. When we lack the needed information (vote extensions), we simply rely + on another correct validator to propose a valid block in other rounds of the current height. + + However, this solution can be attacked by a byzantine node in the network in the following way. + Let us consider the following scenario: + + - all validators in *valseth* send out precommit messages, with vote extensions, + for height *h*, round 0, roughly at the same time, + - all those precommit messages contain non-`nil` precommit votes, which vote for block *b* + - all those precommit messages sent in height *h*, round 0, and all messages sent in + height *h*, round *r > 0* get delayed indefinitely, so, + - all validators in *valseth* keep waiting for enough precommit + messages for height *h*, round 0, needed for deciding in height *h* + - an intermediate (malicious) full node *m* manages to receive block *b*, and gather more than + *2nh/3* precommit messages for height *h*, round 0, + - one way or another, the solution should have either (a) a mechanism for a full node to *tell* + another full node it is late, or (b) a mechanism for a full node to conclude it is late based + on other full nodes' messages; any of these mechanisms should, at the very least, + require the late node receiving the decided block and a commit (not necessarily an extended + commit) for *h*, + - node *m* uses the gathered precommit messages to build a commit for height *h*, round 0, + - in order to convince full nodes that they are late, node *m* either (a) *tells* them they + are late, or (b) shows them it (i.e. *m*) is ahead, by sending them block *b*, along with the + commit for height *h*, round 0, + - all full nodes conclude they are late from *m*'s behavior, and use block *b* and the commit for + height *h*, round 0, to decide on height *h*, and proceed to height *h+1*. + + At this point, *all* correct full nodes, including all correct validators in *valseth+1*, have advanced + to height *h+1* believing they are late, and so, expecting the *hypothetical* leading majority of + validators in *valseth+1* to propose for *h+1*. As a result, the blockchain + grinds to a halt. + A (rather complex) ad-hoc mechanism would need to be carried out by node operators to roll + back all validators to the precommit step of height *h*, round *r*, so that they can regenerate + vote extensions (remember the contents of vote extensions are non-deterministic) and continue execution. + +- **Solution 3.** *Require extended commits to be available at switching time*. + + This one is more involved than all previous solutions, and builds on an idea present in Solution 2: + vote extensions are actually not needed for CometBFT to make progress as long as the + validator is *certain* it is late. + + We define two modes. The first is denoted *catch-up mode*, and CometBFT only calls + `FinalizeBlock` for each height when in this mode. The second is denoted *consensus mode*, in + which the validator considers itself up to date and fully participates in consensus and calls + `PrepareProposal`/`ProcessProposal`, `ExtendVote`, and `VerifyVoteExtension`, before calling + `FinalizeBlock`. + + The catch-up mode does not need vote extension information to make progress, as all it needs is the + decided block at each height to call `FinalizeBlock` and keep the state-machine replication making + progress. The consensus mode, on the other hand, does need vote extension information when + starting every height. + + Validators are in consensus mode by default. When a validator in consensus mode falls behind + for whatever reason, e.g. cases (b), (d), (e), (f), (g), or (h) above, we introduce the following + key safety property: + + - for every height *hp*, a full node *f* in *hp* refuses to switch to catch-up + mode **until** there exists a height *h'* such that: + - *p* has received and (light-client) verified the blocks of + all heights *h*, where *hp ≤ h ≤ h'* + - it has received an extended commit for *h'* and has verified: + - the precommit vote signatures in the extended commit + - the vote extension signatures in the extended commit: each is signed with the same + key as the precommit vote it extends + + If the condition above holds for *hp*, namely receiving a valid sequence of blocks in + *f*'s future, and an extended commit corresponding to the last block in the sequence, then + node *f*: + + - switches to catch-up mode, + - applies all blocks between *hp* and *h'* (calling `FinalizeBlock` only), and + - switches back to consensus mode using the extended commit for *h'* to propose in the rounds of + *h' + 1* where it is the proposer. + + This mechanism, together with the invariant it uses, ensures that the node cannot be attacked by + being fed a block without extensions to make it believe it is late, in a similar way as explained + for Solution 2. + + This solution works as long as the blockchain has vote extensions from genesis, + i.e. it uses ABCI 2.0 from the start. + In contrast, it cannot be used without modifications by a blockchain upgrading + from a previous version of CometBFT that did not implement vote extensions. + In that case, the safety property required to switch to catch-up mode may never hold. + See section [Upgrade Path](#upgrade-path) for further details. + +### Feasibility of the Proposed Solutions + +Solution 0, besides the drawbacks described in the previous section, provides guarantees that are +weaker than the rest. The Application does not have the assurance that more than *2nh/3* vote +extensions will *always* be available when calling `PrepareProposal` at height *h+1*. +This level of guarantees is probably not strong enough for vote extensions to be useful for some +important use cases that motivated them in the first place, e.g., encrypted mempool transactions. + +Solution 1, while being simple in that the changes needed in the current CometBFT codebase would +be rather small, is changing the block format, and would therefore require all blockchains using +ABCI 1.0 or earlier to hard-fork when upgrading to ABCI 2.0. + +Since Solution 2 can be attacked, one might prefer Solution 3, even if it is more involved +to implement. Further, we must elaborate on how we can turn Solution 3, described in abstract +terms in the previous section, into a concrete implementation compatible with the current +CometBFT codebase. + +### Current Limitations and Possible Implementations + +The main limitations affecting the current version of CometBFT are the following. + +- The current version of the blocksync reactor does not use the full + [light client verification][light-client-spec] + algorithm to validate blocks coming from other peers. +- The code being structured into the blocksync and consensus reactors, only switching from the + blocksync reactor to the consensus reactor is supported; switching in the opposite direction is + not supported. Alternatively, the consensus reactor could have a mechanism allowing a late node + to catch up by skipping calls to `PrepareProposal`/`ProcessProposal`, and + `ExtendVote`/`VerifyVoteExtension` and only calling `FinalizeBlock` for each height. + Such a mechanism does not exist at the time of writing this RFC (2023-03-02). + +The blocksync reactor featuring light client verification is among the CometBFT team's current priorities. +So it is best if this RFC does not try to delve into that problem, but just makes sure +its outcomes are compatible with that effort. + +In subsection [Cases to Address](#cases-to-address), we concluded that we can focus on +solving case (h) in theoretical terms. +However, as the current CometBFT version does not yet support switching back to blocksync once a +node has switched to consensus, we need to split case (h) into two cases. When a full node needs to +catch up... + +- **(h.1)** ... it has not switched yet from the blocksync reactor to the consensus reactor, or + +- **(h.2)** ... it has already switched to the consensus reactor. + +This is important in order to discuss the different possible implementations. + +#### Base Implementation: Persist and Propagate Extended Commit History + +In order to circumvent the fact that we cannot switch from the consensus reactor back to blocksync, +rather than just keeping the few most recent extended commits, nodes will need to keep +and gossip a backlog of extended commits so that the consensus reactor can still propose and decide +in out-of-date heights (even if those proposals will be useless). + +The base implementation - which will be part of the first release of ABCI 2.0 - consists in the conservative +approach of persisting in the block store *all* extended commits for which we have also stored +the full block. Currently, when statesync is run at startup, it saves light blocks. +This base implementation does not seek +to receive or persist extended commits for those light blocks as they would not be of any use. + +Then, we modify the blocksync reactor so that peers *always* send requested full blocks together +with the corresponding extended commit in the `BlockResponse` messages. This guarantees that the +block store being reconstructed by blocksync has the same information as that of peers that are +up to date (at least starting from the latest snapshot applied by statesync before starting blocksync). +Thus, blocksync has all the data it requires to switch to the consensus reactor, as long as one of +the following exit conditions are met: + +- The node is still at height 0 (where no commit or extended commit is needed). +- The node has processed at least 1 block in blocksync. +- The node recovered and, after handshaking with the Application, it realizes it had persisted + an extended commit in its block store for the height previous to the one it is to start. + +The second condition is needed in case the node has installed an Application snapshot during statesync. +If that is the case, at the time blocksync starts, the block store only has the data statesync has saved: +light blocks, and no extended commits. +Hence we need to blocksync at least one block from another node, which will be sent with its corresponding extended commit, before we can switch to consensus. + +A chain might be started at a height *hi > 0*, all other heights +*h < hi* being non-existent. In this case, the chain is still considered to be at height 0 before +block *hi* is applied, so the first condition above allows the node to switch to consensus even +if blocksync has not processed any block (which is always the case if all nodes are starting from scratch). + +The third condition is needed to ensure liveness in the case where all validators crash at the same height. +Without the third condition, they all would wait to blocksync at least one block upon recovery. +However, as all validators crashed no further block can be produced and thus blocksync would block forever. + +When a validator falls behind while having already switched to the consensus reactor, a peer node can +simply retrieve the extended commit for the required height from the block store and reconstruct a set of +precommit votes together with their extensions and send them in the form of precommit messages to the +validator falling behind, regardless of whether the peer node holds the extended commit because it +actually participated in that consensus and thus received the precommit messages, or it received the extended commit via a `BlockResponse` message while running blocksync itself. + +This base implementation requires a few changes to the consensus reactor: + +- upon saving the block for a given height in the block store at decision time, save the + corresponding extended commit as well +- in the catch-up mechanism, when a node realizes that another peer is more than 2 heights + behind, it uses the extended commit (rather than the canonical commit as done previously) to + reconstruct the precommit votes with their corresponding extensions + +The changes to the blocksync reactor are more substantial: + +- the `BlockResponse` message is extended to include the extended commit of the same height as + the block included in the response (just as they are stored in the block store) +- structure `bpRequester` is likewise extended to hold the received extended commits coming in + `BlockResponse` messages +- method `PeekTwoBlocks` is modified to also return the extended commit corresponding to the first block +- when successfully verifying a received block, the reactor saves the block along with + its corresponding extended commit in the block store + +The two main drawbacks of this base implementation are: + +- the increased size taken by the block store, in particular with big extensions +- the increased bandwidth taken by the new format of `BlockResponse` + +#### Possible Optimization: Pruning the Extended Commit History + +If we cannot switch from the consensus reactor back to the blocksync reactor we cannot prune the extended commit backlog in the block store without sacrificing the implementation's correctness. The asynchronous +nature of our distributed system model allows a process to fall behind an arbitrary number of +heights, and thus all extended commits need to be kept *just in case* a node that late had +previously switched to the consensus reactor. + +However, there is a possibility to optimize the base implementation. Every time we enter a new height, +we could prune from the block store all extended commits that are more than *d* heights in the past. +Then, we need to handle two new situations, roughly equivalent to cases (h.1) and (h.2) described above. + +- (h.1) A node starts from scratch or recovers after a crash. In this case, we need to modify the + blocksync reactor's base implementation. + - when receiving a `BlockResponse` message, it MUST accept that the extended commit set to `nil`, + - when sending a `BlockResponse` message, if the block store contains the extended commit for that + height, it MUST set it in the message, otherwise it sets it to `nil`, + - the exit conditions used for the base implementation are no longer valid; the only reliable exit + condition now consists in making sure that the last block processed by blocksync was received with + the corresponding commit, and not `nil`; this extended commit will allow the node to switch from + the blocksync reactor to the consensus reactor and immediately act as a proposer if required. +- (h.2) A node already running the consensus reactor falls behind beyond *d* heights. In principle, + the node will be stuck forever as no other node can provide the vote extensions it needs to make + progress (they all have pruned the corresponding extended commit). + However we can manually have the node crash and recover as a workaround. This effectively converts + this case into (h.1). + +Finally, note that it makes sense to pair this optimization with the `retain_height` ABCI parameter. +Whenever we prune blocks from the block store due to `retain_height`, +we also prune the corresponding extended commit. +This is problematic both in (h.1) and (h.2), as a node that falls behind the lowest value +of `retain_height` in the rest of the network will never be able to catch up. +Nevertheless, this problem predates ABCI 2.0, and vote extensions do not make it worse. + +### Upgrade Path + +ABCI 2.0 will be the first version to implement vote extensions. +Upgrading a blockchain to ABCI 2.0 from a previous version MUST be feasible via a coordinated upgrade: +a blockchain upgrading to ABCI 2.0 should not be forced to hard fork (i.e. create a new chain). + +Vote extensions pose an issue for CometBFT upgrades. +Blockchains that perform a coordinated upgrade from ABCI 1.0 to ABCI 2.0 will attempt +to produce the first height running ABCI 2.0 without vote extension data from the previous height. +As explained in previous sections, blockchains running ABCI 2.0 *require* vote extension data in each +[PrepareProposal](https://github.com/cometbft/cometbft/blob/feature/abci++vef/proto/tendermint/abci/types.proto#L134) +call. + +#### New `ConsensusParam` + +To facilitate the upgrade and provide applications a mechanism to require vote extensions, +we introduce a new +[`ConsensusParam`](https://github.com/cometbft/cometbft/blob/38a4cae/proto/tendermint/types/params.proto#L13) +to transition the chain from maintaining no history of vote extensions to requiring vote extensions. +This parameter is an `int64` representing the first height where vote extensions +will be required for votes to be considered valid. + +The initial value of this `ConsensusParam` is 0, +which is also its implicit value in versions prior to ABCI 2.0, +denoting that an extension-enabling height has not been decided yet. +Once the upgrade to ABCI 2.0 has taken place, +the value MAY be set to some height, *he*, +which MUST be higher than the current height of the chain. +From the moment when the `ConsensusParam` > 0, +for all heights *h ≥ he*, the consensus algorithm will +reject any votes that do not have vote extension data as invalid. +Likewise, for all heights *h < he*, any votes that *do* have vote extensions +will be considered an error condition. +Height *he* is somewhat special, as calls to `PrepareProposal` MUST NOT +have vote extension data, but all precommit votes in that height MUST carry a vote extension. +Height *he + 1* is the first height for which `PrepareProposal` MUST have vote +extension data and all precommit votes in that height MUST have a vote extension. + +#### Upgrading and Transitioning to Vote Extensions + +Just after upgrading (via coordinated upgrade) to ABCI 2.0, vote extensions stay disabled, +as the Application needs to decide on a future height to be set for transitioning to vote extensions. +The earliest this can happen is *hu + 1*, where *hu* denotes the upgrade height, +i.e., the height at which all nodes will start when they restart with the upgraded binary. + +Once a node reaches the configured height *he*, the parameter is disallowed from changing. +Vote extensions cannot flip from being required to being optional. +This is enforced by the `ConsensusParam` validation logic. Forcing vote extensions to +be required beyond the configured height simplifies the logic for transitioning +from optional to required since all checks will only need to understand if the +chain *ever* enabled vote extensions in the past. Additionally, the major known +uses cases of vote extensions such as threshold decryption and oracle data will +be *central* components of the applications that use vote extensions. Flipping +vote extensions to be no longer required will fundamentally change the behavior +of the application and is therefore not valuable to these applications. + +Additional discussion and implementation of this upgrade strategy can be found +in GitHub [issue 8453][toggle-vote-extensions]. + +We now explain the changes we need to introduce in key solutions/implementation proposed in previous sections +so that they still work in the presence of an upgrade to ABCI 2.0. +For simplicity, in any conditions comparing a height to *he*, +if *he* is 0 (not set yet) then the condition assumes *he = ∞*. + +#### Changes Required in Solution 3 + +These are the changes needed in Solution 3, as defined in section [Solutions Proposed](#solutions-proposed) +so that it works properly with upgrades. + +First, we need to extend the safety property, which is key to that solution, +to take the agreed extension-enabling height into account. + +The key change is in the switching height *h'*: + +- for every height *hp*, a full node *f* in *hp* refuses to switch to catch-up + mode **until** there exists a height *h'* such that: + - *p* has received and (light-client) verified the blocks of + all heights *h*, where *hp ≤ h ≤ h'* + - if *h' > he* + - it has received an extended commit for *h'* and has verified: + - the precommit vote signatures in the extended commit + - the vote extension signatures in the extended commit: each is signed with the same + key as the precommit vote it extends + +Note that, since the (light-client) verification is the only requirement for all *h' ≤ h*, +the property falls back to the pre-ABCI 2.0 requirements for block sync in those heights. + +#### Changes Required in the Base Implementation + +The base implementation as defined in section +[Base Implementation](#base-implementation-persist-and-propagate-extended-commit-history) +cannot work as such when a blockchain upgrades, and thus it needs the following modifications. + +Firstly, the conditions for switching to consensus listed in section +[Base Implementation](#base-implementation-persist-and-propagate-extended-commit-history) +remain valid, but we need to add a new condition. + +- The node is still at a height *h < he*. + +We have taken the changes required by the base implementation, +initially decribed in section +[Base Implementation](#base-implementation-persist-and-propagate-extended-commit-history), +and adapted them so that +they support upgrading to ABCI 2.0 in the terms described earlier in this section: + +Changes to the consensus reactor: + +- upon saving the block for a given height *h* in the block store at decision time + - if *h ≥ he*, save the corresponding extended commit as well + - if *h < he*, follow the logic implemented prior to ABCI 2.0 +- in the catch-up mechanism, when a node *f* realizes that another peer is at height *hp*, + which is more than 2 heights behind, + - if *hp ≥ he*, *f* uses the extended commit to + reconstruct the precommit votes with their corresponding extensions + - if *hp < he*, *f* uses the canonical commit to reconstruct the precommit votes, + as done for ABCI 1.0 and earlier + +Changes to the blocksync reactor: + +- the `BlockResponse` message is extended to *optionally* include the extended commit of the same height as + the block included in the response (just as they are stored in the block store) +- structure `bpRequester` is likewise extended to *optionally* hold received extended commits coming in + `BlockResponse` messages +- method `PeekTwoBlocks` is modified in the following way + - if the first block's height *h ≥ he*, it returns the block together with the extended commit corresponding to the first block + - if the first block's height *h < he*, it returns the block and `nil` as extended commit +- when successfully verifying a received block, + - if the block's height *h ≥ he*, the reactor saves the block, + along with its corresponding extended commit in the block store + - if the block's height *h < he*, the reactor saves the block in the block store, + and `nil` as extended commit + +### Formalization Work + +A formalization work to show or prove the correctness of the different use cases and solutions +presented here (and any other that may be found) needs to be carried out. +A question that needs a precise answer is how many extended commits (one?, two?) a node needs +to keep in persistent memory when implementing Solution 3 described above without CometBFT's +current limitations. +Another important invariant we need to prove formally is that the set of vote extensions +required to make progress will always be held somewhere in the network. + +## References + +- [ABCI 0.17.0 specification][abci-0-17-0] +- [ABCI 1.0 specification][abci-1-0] +- [ABCI 2.0 specification][abci-2-0] +- [Light client verification][light-client-spec] +- [Empty vote extensions issue](https://github.com/tendermint/tendermint/issues/8174) +- [Toggle vote extensions issue][toggle-vote-extensions] + +[abci-0-17-0]: https://github.com/cometbft/cometbft/blob/v0.34.x/spec/abci/README.md +[abci-1-0]: https://github.com/cometbft/cometbft/blob/v0.37.x/spec/abci/README.md +[abci-2-0]: https://github.com/cometbft/cometbft/blob/v0.38.x/spec/abci/README.md +[light-client-spec]: https://github.com/cometbft/cometbft/blob/v0.38.x/spec/light-client/README.md +[toggle-vote-extensions]: https://github.com/tendermint/tendermint/issues/8453 diff --git a/cometbft/v0.38/docs/rfc/rfc-template.md b/cometbft/v0.38/docs/rfc/rfc-template.md new file mode 100644 index 00000000..b3f40477 --- /dev/null +++ b/cometbft/v0.38/docs/rfc/rfc-template.md @@ -0,0 +1,35 @@ +# RFC {RFC-NUMBER}: {TITLE} + +## Changelog + +- {date}: {changelog} + +## Abstract + +> A brief high-level synopsis of the topic of discussion for this RFC, ideally +> just a few sentences. This should help the reader quickly decide whether the +> rest of the discussion is relevant to their interest. + +## Background + +> Any context or orientation needed for a reader to understand and participate +> in the substance of the Discussion. If necessary, this section may include +> links to other documentation or sources rather than restating existing +> material, but should provide enough detail that the reader can tell what they +> need to read to be up-to-date. + +### References + +> Links to external materials needed to follow the discussion may be added here. +> +> In addition, if the discussion in a request for comments leads to any design +> decisions, it may be helpful to add links to the ADR documents here after the +> discussion has settled. + +## Discussion + +> This section contains the core of the discussion. +> +> There is no fixed format for this section, but ideally changes to this +> section should be updated before merging to reflect any discussion that took +> place on the PR that made those changes. diff --git a/cometbft/v0.38/docs/rfc/tendermint-core/README.md b/cometbft/v0.38/docs/rfc/tendermint-core/README.md new file mode 100644 index 00000000..2a164b2c --- /dev/null +++ b/cometbft/v0.38/docs/rfc/tendermint-core/README.md @@ -0,0 +1,41 @@ +--- +order: 1 +parent: + order: false +--- + +# Tendermint Core Requests for Comments + +This document serves as a historical reference for all RFCs that were logged +during the development of Tendermint Core. + +This list is frozen as-is, and new RFCs should be added [here](../). + +## Table of Contents + +- [RFC-000: P2P Roadmap](./rfc-000-p2p-roadmap.rst) +- [RFC-001: Storage Engines](./rfc-001-storage-engine.rst) +- [RFC-002: Interprocess Communication](./rfc-002-ipc-ecosystem.md) +- [RFC-003: Performance Taxonomy](./rfc-003-performance-questions.md) +- [RFC-004: E2E Test Framework Enhancements](./rfc-004-e2e-framework.rst) +- [RFC-005: Event System](./rfc-005-event-system.rst) +- [RFC-006: Event Subscription](./rfc-006-event-subscription.md) +- [RFC-007: Deterministic Proto Byte Serialization](./rfc-007-deterministic-proto-bytes.md) +- [RFC-008: Don't Panic](./rfc-008-do-not-panic.md) +- [RFC-009: Consensus Parameter Upgrades](./rfc-009-consensus-parameter-upgrades.md) +- [RFC-010: P2P Light Client](./rfc-010-p2p-light-client.rst) +- [RFC-011: Delete Gas](./rfc-011-delete-gas.md) +- [RFC-012: Event Indexing Revisited](./rfc-012-custom-indexing.md) +- [RFC-013: ABCI++](./rfc-013-abci++.md) +- [RFC-014: Semantic Versioning](./rfc-014-semantic-versioning.md) +- [RFC-015: ABCI++ Tx Mutation](./rfc-015-abci++-tx-mutation.md) +- [RFC-016: Node Architecture](./rfc-016-node-architecture.md) +- [RFC-017: ABCI++ Vote Extension Propagation](./rfc-017-abci++-vote-extension-propag.md) +- [RFC-018: BLS Signature Aggregation Exploration](./rfc-018-bls-agg-exploration.md) +- [RFC-019: Configuration File Versioning](./rfc-019-config-version.md) +- [RFC-020: Onboarding Projects](./rfc-020-onboarding-projects.rst) +- [RFC-021: The Future of the Socket Protocol](./rfc-021-socket-protocol.md) +- [RFC-023: Semi-permanent Testnet](./rfc-023-semi-permanent-testnet.md) +- [RFC-024: Block Structure Consolidation](./rfc-024-block-structure-consolidation.md) +- [RFC-025: Application Defined Transaction Storage](./rfc-025-support-app-side-mempool.md) +- [RFC-027: P2P Message Bandwidth Report](./rfc-027-p2p-message-bandwidth-report.md) diff --git a/cometbft/v0.38/docs/rfc/tendermint-core/images/abci++.png b/cometbft/v0.38/docs/rfc/tendermint-core/images/abci++.png new file mode 100644 index 00000000..d5146f99 Binary files /dev/null and b/cometbft/v0.38/docs/rfc/tendermint-core/images/abci++.png differ diff --git a/cometbft/v0.38/docs/rfc/tendermint-core/images/abci.png b/cometbft/v0.38/docs/rfc/tendermint-core/images/abci.png new file mode 100644 index 00000000..10039ab5 Binary files /dev/null and b/cometbft/v0.38/docs/rfc/tendermint-core/images/abci.png differ diff --git a/cometbft/v0.38/docs/rfc/tendermint-core/images/node-dependency-tree.svg b/cometbft/v0.38/docs/rfc/tendermint-core/images/node-dependency-tree.svg new file mode 100644 index 00000000..6d95e0e1 --- /dev/null +++ b/cometbft/v0.38/docs/rfc/tendermint-core/images/node-dependency-tree.svg @@ -0,0 +1,3 @@ + + +
Node
Node
Statesync
Statesync
Blocksync
Blocksync
Consensus
Consensus
Mempool
Mempool
Evidence
Evidence
Block Executor
Block Executor
Blockchain
Blockchain
Evidence
Evidence
PEX
PEX
Peer Store
Peer Store
Peer Networking
Peer Networking
RPC External
RPC External
ABCI Layer
ABCI Layer
Events System
Events System
RPC Internal
RPC Internal
Viewer does not support full SVG 1.1
\ No newline at end of file diff --git a/cometbft/v0.38/docs/rfc/tendermint-core/images/receive-rate-all.png b/cometbft/v0.38/docs/rfc/tendermint-core/images/receive-rate-all.png new file mode 100644 index 00000000..b28a83f6 Binary files /dev/null and b/cometbft/v0.38/docs/rfc/tendermint-core/images/receive-rate-all.png differ diff --git a/cometbft/v0.38/docs/rfc/tendermint-core/images/send-rate-all.png b/cometbft/v0.38/docs/rfc/tendermint-core/images/send-rate-all.png new file mode 100644 index 00000000..803aca21 Binary files /dev/null and b/cometbft/v0.38/docs/rfc/tendermint-core/images/send-rate-all.png differ diff --git a/cometbft/v0.38/docs/rfc/tendermint-core/images/top-3-percent-receive.png b/cometbft/v0.38/docs/rfc/tendermint-core/images/top-3-percent-receive.png new file mode 100644 index 00000000..39c5a2d0 Binary files /dev/null and b/cometbft/v0.38/docs/rfc/tendermint-core/images/top-3-percent-receive.png differ diff --git a/cometbft/v0.38/docs/rfc/tendermint-core/images/top-3-percent-send.png b/cometbft/v0.38/docs/rfc/tendermint-core/images/top-3-percent-send.png new file mode 100644 index 00000000..61f7f6e6 Binary files /dev/null and b/cometbft/v0.38/docs/rfc/tendermint-core/images/top-3-percent-send.png differ diff --git a/cometbft/v0.38/docs/rfc/tendermint-core/rfc-000-p2p-roadmap.rst b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-000-p2p-roadmap.rst new file mode 100644 index 00000000..4f3e868c --- /dev/null +++ b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-000-p2p-roadmap.rst @@ -0,0 +1,316 @@ +==================== +RFC 000: P2P Roadmap +==================== + +Changelog +--------- + +- 2021-08-20: Completed initial draft and distributed via a gist +- 2021-08-25: Migrated as an RFC and changed format + +Abstract +-------- + +This document discusses the future of peer network management in Tendermint, with +a particular focus on features, semantics, and a proposed roadmap. +Specifically, we consider libp2p as a tool kit for implementing some fundamentals. + +Background +---------- + +For the 0.35 release cycle the switching/routing layer of Tendermint was +replaced. This work was done "in place," and produced a version of Tendermint +that was backward-compatible and interoperable with previous versions of the +software. While there are new p2p/peer management constructs in the new +version (e.g. ``PeerManager`` and ``Router``), the main effect of this change +was to simplify the ways that other components within Tendermint interacted with +the peer management layer, and to make it possible for higher-level components +(specifically the reactors), to be used and tested more independently. + +This refactoring, which was a major undertaking, was entirely necessary to +enable areas for future development and iteration on this aspect of +Tendermint. There are also a number of potential user-facing features that +depend heavily on the p2p layer: additional transport protocols, transport +compression, improved resilience to network partitions. These improvements to +modularity, stability, and reliability of the p2p system will also make +ongoing maintenance and feature development easier in the rest of Tendermint. + +Critique of Current Peer-to-Peer Infrastructure +--------------------------------------- + +The current (refactored) P2P stack is an improvement on the previous iteration +(legacy), but as of 0.35, there remains room for improvement in the design and +implementation of the P2P layer. + +Some limitations of the current stack include: + +- heavy reliance on buffering to avoid backups in the flow of components, + which is fragile to maintain and can lead to unexpected memory usage + patterns and forces the routing layer to make decisions about when messages + should be discarded. + +- the current p2p stack relies on convention (rather than the compiler) to + enforce the API boundaries and conventions between reactors and the router, + making it very easy to write "wrong" reactor code or introduce a bad + dependency. + +- the current stack is probably more complex and difficult to maintain because + the legacy system must coexist with the new components in 0.35. When the + legacy stack is removed there are some simple changes that will become + possible and could reduce the complexity of the new system. (e.g. `#6598 + `_.) + +- the current stack encapsulates a lot of information about peers, and makes it + difficult to expose that information to monitoring/observability tools. This + general opacity also makes it difficult to interact with the peer system + from other areas of the code base (e.g. tests, reactors). + +- the legacy stack provided some control to operators to force the system to + dial new peers or seed nodes or manipulate the topology of the system _in + situ_. The current stack can't easily provide this, and while the new stack + may have better behavior, it does leave operators hands tied. + +Some of these issues will be resolved early in the 0.36 cycle, with the +removal of the legacy components. + +The 0.36 release also provides the opportunity to make changes to the +protocol, as the release will not be compatible with previous releases. + +Areas for Development +--------------------- + +These sections describe features that may make sense to include in a Phase 2 of +a P2P project. + +Internal Message Passing +~~~~~~~~~~~~~~~~~~~~~~~~ + +Currently, there's no provision for intranode communication using the P2P +layer, which means when two reactors need to interact with each other they +have to have dependencies on each other's interfaces, and +initialization. Changing these interactions (e.g. transitions between +blocksync and consensus) from procedure calls to message passing. + +This is a relatively simple change and could be implemented with the following +components: + +- a constant to represent "local" delivery as the ``To`` field on + ``p2p.Envelope``. + +- special path for routing local messages that doesn't require message + serialization (protobuf marshalling/unmarshaling). + +Adding these semantics, particularly if in conjunction with synchronous +semantics provides a solution to dependency graph problems currently present +in the Tendermint codebase, which will simplify development, make it possible +to isolate components for testing. + +Eventually, this will also make it possible to have a logical Tendermint node +running in multiple processes or in a collection of containers, although the +usecase of this may be debatable. + +Synchronous Semantics (Paired Request/Response) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In the current system, all messages are sent with fire-and-forget semantics, +and there's no coupling between a request sent via the p2p layer, and a +response. These kinds of semantics would simplify the implementation of +state and block sync reactors, and make intra-node message passing more +powerful. + +For some interactions, like gossiping transactions between the mempools of +different nodes, fire-and-forget semantics make sense, but for other +operations the missing link between requests/responses leads to either +inefficiency when a node fails to respond or becomes unavailable, or code that +is just difficult to follow. + +To support this kind of work, the protocol would need to accommodate some kind +of request/response ID to allow identifying out-of-order responses over a +single connection. Additionally, expanded the programming model of the +``p2p.Channel`` to accommodate some kind of _future_ or similar paradigm to +make it viable to write reactor code without needing for the reactor developer +to wrestle with lower level concurrency constructs. + + +Timeout Handling (QoS) +~~~~~~~~~~~~~~~~~~~~~~ + +Currently, all timeouts, buffering, and QoS features are handled at the router +layer, and the reactors are implemented in ways that assume/require +asynchronous operation. This both increases the required complexity at the +routing layer, and means that misbehavior at the reactor level is difficult to +detect or attribute. Additionally, the current system provides three main +parameters to control quality of service: + +- buffer sizes for channels and queues. + +- priorities for channels + +- queue implementation details for shedding load. + +These end up being quite coarse controls, and changing the settings are +difficult because as the queues and channels are able to buffer large numbers +of messages it can be hard to see the impact of a given change, particularly +in our extant test environment. In general, we should endeavor to: + +- set real timeouts, via contexts, on most message send operations, so that + senders rather than queues can be responsible for timeout + logic. Additionally, this will make it possible to avoid sending messages + during shutdown. + +- reduce (to the greatest extent possible) the amount of buffering in + channels and the queues, to more readily surface backpressure and reduce the + potential for buildup of stale messages. + +Stream Based Connection Handling +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Currently the transport layer is message based, which makes sense from a +mental model of how the protocol works, but makes it more difficult to +implement transports and connection types, as it forces a higher level view of +the connection and interaction which makes it harder to implement for novel +transport types and makes it more likely that message-based caching and rate +limiting will be implemented at the transport layer rather than at a more +appropriate level. + +The transport then, would be responsible for negotiating the connection and the +handshake and otherwise behave like a socket/file descriptor with ``Read`` and +``Write`` methods. + +While this was included in the initial design for the new P2P layer, it may be +obviated entirely if the transport and peer layer is replaced with libp2p, +which is primarily stream based. + +Service Discovery +~~~~~~~~~~~~~~~~~ + +In the current system, Tendermint assumes that all nodes in a network are +largely equivalent, and nodes tend to be "chatty" making many requests of +large numbers of peers and waiting for peers to (hopefully) respond. While +this works and has allowed Tendermint to get to a certain point, this both +produces a theoretical scaling bottle neck and makes it harder to test and +verify components of the system. + +In addition to peer's identity and connection information, peers should be +able to advertise a number of services or capabilities, and node operators or +developers should be able to specify peer capability requirements (e.g. target +at least -percent of peers with capability.) + +These capabilities may be useful in selecting peers to send messages to, it +may make sense to extend Tendermint's message addressing capability to allow +reactors to send messages to groups of peers based on role rather than only +allowing addressing to one or all peers. + +Having a good service discovery mechanism may pair well with the synchronous +semantics (request/response) work, as it allows reactors to "make a request of +a peer with capability and wait for the response," rather force the +reactors to need to track the capabilities or state of specific peers. + +Solutions +--------- + +Continued Homegrown Implementation +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The current peer system is homegrown and is conceptually compatible with the +needs of the project, and while there are limitations to the system, the p2p +layer is not (currently as of 0.35) a major source of bugs or friction during +development. + +However, the current implementation makes a number of allowances for +interoperability, and there are a collection of iterative improvements that +should be considered in the next couple of releases. To maintain the current +implementation, upcoming work would include: + +- change the ``Transport`` mechanism to facilitate easier implementations. + +- implement different ``Transport`` handlers to be able to manage peer + connections using different protocols (e.g. QUIC, etc.) + +- entirely remove the constructs and implementations of the legacy peer + implementation. + +- establish and enforce clearer chains of responsibility for connection + establishment (e.g. handshaking, setup,) which is currently shared between + three components. + +- report better metrics regarding the into the state of peers and network + connectivity, which are opaque outside of the system. This is constrained at + the moment as a side effect of the split responsibility for connection + establishment. + +- extend the PEX system to include service information so that nodes in the + network weren't necessarily homogeneous. + +While maintaining a bespoke peer management layer would seem to distract from +development of core functionality, the truth is that (once the legacy code is +removed,) the scope of the peer layer is relatively small from a maintenance +perspective, and having control at this layer might actually afford the +project with the ability to more rapidly iterate on some features. + +LibP2P +~~~~~~ + +LibP2P provides components that, approximately, account for the +``PeerManager`` and ``Transport`` components of the current (new) P2P +stack. The Go APIs seem reasonable, and being able to externalize the +implementation details of peer and connection management seems like it could +provide a lot of benefits, particularly in supporting a more active ecosystem. + +In general the API provides the kind of stream-based, multi-protocol +supporting, and idiomatic baseline for implementing a peer layer. Additionally +because it handles peer exchange and connection management at a lower +level, by using libp2p it'd be possible to remove a good deal of code in favor +of just using libp2p. Having said that, Tendermint's P2P layer covers a +greater scope (e.g. message routing to different peers) and that layer is +something that Tendermint might want to retain. + +The are a number of unknowns that require more research including how much of +a peer database the Tendermint engine itself needs to maintain, in order to +support higher level operations (consensus, statesync), but it might be the +case that our internal systems need to know much less about peers than +otherwise specified. Similarly, the current system has a notion of peer +scoring that cannot be communicated to libp2p, which may be fine as this is +only used to support peer exchange (PEX,) which would become a property libp2p +and not expressed in it's current higher-level form. + +In general, the effort to switch to libp2p would involve: + +- timing it during an appropriate protocol-breaking window, as it doesn't seem + viable to support both libp2p *and* the current p2p protocol. + +- providing some in-memory testing network to support the use case that the + current ``p2p.MemoryNetwork`` provides. + +- re-homing the ``p2p.Router`` implementation on top of libp2p components to + be able to maintain the current reactor implementations. + +Open question include: + +- how much local buffering should we be doing? It sort of seems like we should + figure out what the expected behavior is for libp2p for QoS-type + functionality, and if our requirements mean that we should be implementing + this on top of things ourselves? + +- if Tendermint was going to use libp2p, how would libp2p's stability + guarantees (protocol, etc.) impact/constrain Tendermint's stability + guarantees? + +- what kind of introspection does libp2p provide, and to what extend would + this change or constrain the kind of observability that Tendermint is able + to provide? + +- how do efforts to select "the best" (healthy, close, well-behaving, etc.) + peers work out if Tendermint is not maintaining a local peer database? + +- would adding additional higher level semantics (internal message passing, + request/response pairs, service discovery, etc.) facilitate removing some of + the direct linkages between constructs/components in the system and reduce + the need for Tendermint nodes to maintain state about its peers? + +References +---------- + +- `Tracking Ticket for P2P Refactor Project `_ +- `ADR 61: P2P Refactor Scope <../architecture/adr-061-p2p-refactor-scope.md>`_ +- `ADR 62: P2P Architecture and Abstraction <../architecture/adr-062-p2p-architecture.md>`_ diff --git a/cometbft/v0.38/docs/rfc/tendermint-core/rfc-001-storage-engine.rst b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-001-storage-engine.rst new file mode 100644 index 00000000..560e8a8b --- /dev/null +++ b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-001-storage-engine.rst @@ -0,0 +1,179 @@ +=========================================== +RFC 001: Storage Engines and Database Layer +=========================================== + +Changelog +--------- + +- 2021-04-19: Initial Draft (gist) +- 2021-09-02: Migrated to RFC folder, with some updates + +Abstract +-------- + +The aspect of Tendermint that's responsible for persistence and storage (often +"the database" internally) represents a bottle neck in the architecture of the +platform, that the 0.36 release presents a good opportunity to correct. The +current storage engine layer provides a great deal of flexibility that is +difficult for users to leverage or benefit from, while also making it harder +for Tendermint Core developers to deliver improvements on storage engine. This +RFC discusses the possible improvements to this layer of the system. + +Background +---------- + +Tendermint has a very thin common wrapper that makes Tendermint itself +(largely) agnostic to the data storage layer (within the realm of the popular +key-value/embedded databases.) This flexibility is not particularly useful: +the benefits of a specific database engine in the context of Tendermint is not +particularly well understood, and the maintenance burden for multiple backends +is not commensurate with the benefit provided. Additionally, because the data +storage layer is handled generically, and most tests run with an in-memory +framework, it's difficult to take advantage of any higher-level features of a +database engine. + +Ideally, developers within Tendermint will be able to interact with persisted +data via an interface that can function, approximately like an object +store, and this storage interface will be able to accommodate all existing +persistence workloads (e.g. block storage, local peer management information +like the "address book", crash-recovery log like the WAL.) In addition to +providing a more ergonomic interface and new semantics, by selecting a single +storage engine tendermint can use native durability and atomicity features of +the storage engine and simplify its own implementations. + +Data Access Patterns +~~~~~~~~~~~~~~~~~~~~ + +Tendermint's data access patterns have the following characteristics: + +- aggregate data size often exceeds memory. + +- data is rarely mutated after it's written for most data (e.g. blocks), but + small amounts of working data is persisted by nodes and is frequently + mutated (e.g. peer information, validator information.) + +- read patterns can be quite random. + +- crash resistance and crash recovery, provided by write-ahead-logs (in + consensus, and potentially for the mempool) should allow the system to + resume work after an unexpected shut down. + +Project Goals +~~~~~~~~~~~~~ + +As we think about replacing the current persistence layer, we should consider +the following high level goals: + +- drop dependencies on storage engines that have a CGo dependency. + +- encapsulate data format and data storage from higher-level services + (e.g. reactors) within tendermint. + +- select a storage engine that does not incur any additional operational + complexity (e.g. database should be embedded.) + +- provide database semantics with sufficient ACID, snapshots, and + transactional support. + +Open Questions +~~~~~~~~~~~~~~ + +The following questions remain: + +- what kind of data-access concurrency does tendermint require? + +- would tendermint users SDK/etc. benefit from some shared database + infrastructure? + + - In earlier conversations it seemed as if the SDK has selected Badger and + RocksDB for their storage engines, and it might make sense to be able to + (optionally) pass a handle to a Badger instance between the libraries in + some cases. + +- what are typical data sizes, and what kinds of memory sizes can we expect + operators to be able to provide? + +- in addition to simple persistence, what kind of additional semantics would + tendermint like to enjoy (e.g. transactional semantics, unique constraints, + indexes, in-place-updates, etc.)? + +Decision Framework +~~~~~~~~~~~~~~~~~~ + +Given the constraint of removing the CGo dependency, the decision is between +"badger" and "boltdb" (in the form of the etcd/CoreOS fork,) as low level. On +top of this and somewhat orthogonally, we must also decide on the interface to +the database and how the larger application will have to interact with the +database layer. Users of the data layer shouldn't ever need to interact with +raw byte slices from the database, and should mostly have the experience of +interacting with Go-types. + +Badger is more consistently developed and has a broader feature set than +Bolt. At the same time, Badger is likely more memory intensive and may have +more overhead in terms of open file handles given it's model. At first glance, +Badger is the obvious choice: it's actively developed and it has a lot of +features that could be useful. Bolt is not without some benefits: it's stable +and is maintained by the etcd folks, it's simpler model (single memory mapped +file, etc,) may be easier to reason about. + +I propose that we consider the following specific questions about storage +engines: + +- does Badger's evolving development, which may result in data file format + changes in the future, and could restrict our access to using the latest + version of the library between major upgrades, present a problem? + +- do we do we have goals/concerns about memory footprint that Badger may + prevent us from hitting, particularly as data sets grow over time? + +- what kind of additional tooling might we need/like to build (dump/restore, + etc.)? + +- do we want to run unit/integration tests against a data files on disk rather + than relying exclusively on the memory database? + +Project Scope +~~~~~~~~~~~~~ + +This project will consist of the following aspects: + +- selecting a storage engine, and modifying the tendermint codebase to + disallow any configuration of the storage engine outside of the tendermint. + +- remove the dependency on the current tm-db interfaces and replace with some + internalized, safe, and ergonomic interface for data persistence with all + required database semantics. + +- update core tendermint code to use the new interface and data tools. + +Next Steps +~~~~~~~~~~ + +- circulate the RFC, and discuss options with appropriate stakeholders. + +- write brief ADR to summarize decisions around technical decisions reached + during the RFC phase. + +References +---------- + +- `bolddb `_ +- `badger `_ +- `badgerdb overview `_ +- `botldb overview `_ +- `boltdb vs badger `_ +- `bolthold `_ +- `badgerhold `_ +- `Pebble `_ +- `SDK Issue Regarding IVAL `_ +- `SDK Discussion about SMT/IVAL `_ + +Discussion +---------- + +- All things being equal, my tendency would be to use badger, with badgerhold + (if that makes sense) for its ergonomics and indexing capabilities, which + will require some small selection of wrappers for better write transaction + support. This is a weakly held tendency/belief and I think it would be + useful for the RFC process to build consensus (or not) around this basic + assumption. diff --git a/cometbft/v0.38/docs/rfc/tendermint-core/rfc-002-ipc-ecosystem.md b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-002-ipc-ecosystem.md new file mode 100644 index 00000000..d3eba00b --- /dev/null +++ b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-002-ipc-ecosystem.md @@ -0,0 +1,420 @@ +# RFC 002: Interprocess Communication (IPC) in Tendermint + +## Changelog + +- 08-Sep-2021: Initial draft (@creachadair). + + +## Abstract + +Communication in Tendermint among consensus nodes, applications, and operator +tools all use different message formats and transport mechanisms. In some +cases there are multiple options. Having all these options complicates both the +code and the developer experience, and hides bugs. To support a more robust, +trustworthy, and usable system, we should document which communication paths +are essential, which could be removed or reduced in scope, and what we can +improve for the most important use cases. + +This document proposes a variety of possible improvements of varying size and +scope. Specific design proposals should get their own documentation. + + +## Background + +The Tendermint state replication engine has a complex IPC footprint. + +1. Consensus nodes communicate with each other using a networked peer-to-peer + message-passing protocol. + +2. Consensus nodes communicate with the application whose state is being + replicated via the [Application BlockChain Interface (ABCI)][abci]. + +3. Consensus nodes export a network-accessible [RPC service][rpc-service] to + support operations (bootstrapping, debugging) and synchronization of [light clients][light-client]. + This interface is also used by the [`tendermint` CLI][tm-cli]. + +4. Consensus nodes export a gRPC service exposing a subset of the methods of + the RPC service described by (3). This was intended to simplify the + implementation of tools that already use gRPC to communicate with an + application (via the Cosmos SDK), and wanted to also talk to the consensus + node without implementing yet another RPC protocol. + + The gRPC interface to the consensus node has been deprecated and is slated + for removal in the forthcoming Tendermint v0.36 release. + +5. Consensus nodes may optionally communicate with a "remote signer" that holds + a validator key and can provide public keys and signatures to the consensus + node. One of the stated goals of this configuration is to allow the signer + to be run on a private network, separate from the consensus node, so that a + compromise of the consensus node from the public network would be less + likely to expose validator keys. + +## Discussion: Transport Mechanisms + +### Remote Signer Transport + +A remote signer communicates with the consensus node in one of two ways: + +1. "Raw": Using a TCP or Unix-domain socket which carries varint-prefixed + protocol buffer messages. In this mode, the consensus node is the server, + and the remote signer is the client. + + This mode has been deprecated, and is intended to be removed. + +2. gRPC: This mode uses the same protobuf messages as "Raw" node, but uses a + standard encrypted gRPC HTTP/2 stub as the transport. In this mode, the + remote signer is the server and the consensus node is the client. + + +### ABCI Transport + +In ABCI, the _application_ is the server, and the Tendermint consensus engine +is the client. Most applications implement the server using the [Cosmos SDK][cosmos-sdk], +which handles low-level details of the ABCI interaction and provides a +higher-level interface to the rest of the application. The SDK is written in Go. + +Beneath the SDK, the application communicates with Tendermint core in one of +two ways: + +- In-process direct calls (for applications written in Go and compiled against + the Tendermint code). This is an optimization for the common case where an + application is written in Go, to save on the overhead of marshaling and + unmarshaling requests and responses within the same process: + [`abci/client/local_client.go`][local-client] + +- A custom remote procedure protocol built on wire-format protobuf messages + using a socket (the "socket protocol"): [`abci/server/socket_server.go`][socket-server] + +The SDK also provides a [gRPC service][sdk-grpc] accessible from outside the +application, allowing transactions to be broadcast to the network, look up +transactions, and simulate transaction costs. + + +### RPC Transport + +The consensus node RPC service allows callers to query consensus parameters +(genesis data, transactions, commits), node status (network info, health +checks), application state (abci_query, abci_info), mempool state, and other +attributes of the node and its application. The service also provides methods +allowing transactions and evidence to be injected ("broadcast") into the +blockchain. + +The RPC service is exposed in several ways: + +- HTTP GET: Queries may be sent as URI parameters, with method names in the path. + +- HTTP POST: Queries may be sent as JSON-RPC request messages in the body of an + HTTP POST request. The server uses a custom implementation of JSON-RPC that + is not fully compatible with the [JSON-RPC 2.0 spec][json-rpc], but handles + the common cases. + +- Websocket: Queries may be sent as JSON-RPC request messages via a websocket. + This transport uses more or less the same JSON-RPC plumbing as the HTTP POST + handler. + + The websocket endpoint also includes three methods that are _only_ exported + via websocket, which appear to support event subscription. + +- gRPC: A subset of queries may be issued in protocol buffer format to the gRPC + interface described above under (4). As noted, this endpoint is deprecated + and will be removed in v0.36. + +### Opportunities for Simplification + +**Claim:** There are too many IPC mechanisms. + +The preponderance of ABCI usage is via the Cosmos SDK, which means the +application and the consensus node are compiled together into a single binary, +and the consensus node calls the ABCI methods of the application directly as Go +functions. + +We also need a true IPC transport to support ABCI applications _not_ written in +Go. There are also several known applications written in Rust, for example +(including [Anoma](https://github.com/anoma/anoma), Penumbra, +[Oasis](https://github.com/oasisprotocol/oasis-core), Twilight, and +[Nomic](https://github.com/nomic-io/nomic)). Ideally we will have at most one +such transport "built-in": More esoteric cases can be handled by a custom proxy. +Pragmatically, gRPC is probably the right choice here. + +The primary consumers of the multi-headed "RPC service" today are the light +client and the `tendermint` command-line client. There is probably some local +use via curl, but I expect that is mostly ad hoc. Ethan reports that nodes are +often configured with the ports to the RPC service blocked, which is good for +security but complicates use by the light client. + +### Context: Remote Signer Issues + +Since the remote signer needs a secure communication channel to exchange keys +and signatures, and is expected to run truly remotely from the node (i.e., on a +separate physical server), there is not a whole lot we can do here. We should +finish the deprecation and removal of the "raw" socket protocol between the +consensus node and remote signers, but the use of gRPC is appropriate. + +The main improvement we can make is to simplify the implementation quite a bit, +once we no longer need to support both "raw" and gRPC transports. + +### Context: ABCI Issues + +In the original design of ABCI, the presumption was that all access to the +application should be mediated by the consensus node. The idea is that outside +access could change application state and corrupt the consensus process, which +relies on the application to be deterministic. Of course, even without outside +access an application could behave nondeterministically, but allowing other +programs to send it requests was seen as courting trouble. + +Conversely, users noted that most of the time, tools written for a particular +application don't want to talk to the consensus module directly. The +application "owns" the state machine the consensus engine is replicating, so +tools that care about application state should talk to the application. +Otherwise, they would have to bake in knowledge about Tendermint (e.g., its +interfaces and data structures) just because of the mediation. + +For clients to talk directly to the application, however, there is another +concern: The consensus node is the ABCI _client_, so it is inconvenient for the +application to "push" work into the consensus module via ABCI itself. The +current implementation works around this by calling the consensus node's RPC +service, which exposes an `ABCIQuery` kitchen-sink method that allows the +application a way to poke ABCI messages in the other direction. + +Without this RPC method, you could work around this (at least in principle) by +having the consensus module "poll" the application for work that needs done, +but that has unsatisfactory implications for performance and robustness, as +well as being harder to understand. + +There has apparently been discussion about trying to make a more bidirectional +communication between the consensus node and the application, but this issue +seems to still be unresolved. + +Another complication of ABCI is that it requires the application (server) to +maintain [four separate connections][abci-conn]: One for "consensus" operations +(BeginBlock, EndBlock, DeliverTx, Commit), one for "mempool" operations, one +for "query" operations, and one for "snapshot" (state synchronization) operations. +The rationale seems to have been that these groups of operations should be able +to proceed concurrently with each other. In practice, it results in a very complex +state management problem to coordinate state updates between the separate streams. +While application authors in Go are mostly insulated from that complexity by the +Cosmos SDK, the plumbing to maintain those separate streams is complicated, hard +to understand, and we suspect it contains concurrency bugs and/or lock contention +issues affecting performance that are subtle and difficult to pin down. + +Even without changing the semantics of any ABCI operations, this code could be +made smaller and easier to debug by separating the management of concurrency +and locking from the IPC transport: If all requests and responses are routed +through one connection, the server can explicitly maintain priority queues for +requests and responses, and make less-conservative decisions about when locks +are (or aren't) required to synchronize state access. With independent queues, +the server must lock conservatively, and no optimistic scheduling is practical. + +This would be a tedious implementation change, but should be achievable without +breaking any of the existing interfaces. More importantly, it could potentially +address a lot of difficult concurrency and performance problems we currently +see anecdotally but have difficultly isolating because of how intertwined these +separate message streams are at runtime. + +TODO: Impact of ABCI++ for this topic? + +### Context: RPC Issues + +The RPC system serves several masters, and has a complex surface area. I +believe there are some improvements that can be exposed by separating some of +these concerns. + +The Tendermint light client currently uses the RPC service to look up blocks +and transactions, and to forward ABCI queries to the application. The light +client proxy uses the RPC service via a websocket. The Cosmos IBC relayer also +uses the RPC service via websocket to watch for transaction events, and uses +the `ABCIQuery` method to fetch information and proofs for posted transactions. + +Some work is already underway toward using P2P message passing rather than RPC +to synchronize light client state with the rest of the network. IBC relaying, +however, requires access to the event system, which is currently not accessible +except via the RPC interface. Event subscription _could_ be exposed via P2P, +but that is a larger project since it adds P2P communication load, and might +thus have an impact on the performance of consensus. + +If event subscription can be moved into the P2P network, we could entirely +remove the websocket transport, even for clients that still need access to the +RPC service. Until then, we may still be able to reduce the scope of the +websocket endpoint to _only_ event subscription, by moving uses of the RPC +server as a proxy to ABCI over to the gRPC interface. + +Having the RPC server still makes sense for local bootstrapping and operations, +but can be further simplified. Here are some specific proposals: + +- Remove the HTTP GET interface entirely. + +- Simplify JSON-RPC plumbing to remove unnecessary reflection and wrapping. + +- Remove the gRPC interface (this is already planned for v0.36). + +- Separate the websocket interface from the rest of the RPC service, and + restrict it to only event subscription. + + Eventually we should try to emove the websocket interface entirely, but we + will need to revisit that (probably in a new RFC) once we've done some of the + easier things. + +These changes would preserve the ability of operators to issue queries with +curl (but would require using JSON-RPC instead of URI parameters). That would +be a little less user-friendly, but for a use case that should not be that +prevalent. + +These changes would also preserve compatibility with existing JSON-RPC based +code paths like the `tendermint` CLI and the light client (even ahead of +further work to remove that dependency). + +**Design goal:** An operator should be able to disable non-local access to the +RPC server on any node in the network without impairing the ability of the +network to function for service of state replication, including light clients. + +**Design principle:** All communication required to implement and monitor the +consensus network should use P2P, including the various synchronizations. + +### Options for ABCI Transport + +The majority of current usage is in Go, and the majority of that is mediated by +the Cosmos SDK, which uses the "direct call" interface. There is probably some +opportunity to clean up the implementation of that code, notably by inverting +which interface is at the "top" of the abstraction stack (currently it acts +like an RPC interface, and escape-hatches into the direct call). However, this +general approach works fine and doesn't need to be fundamentally changed. + +For applications _not_ written in Go, the two remaining options are the +"socket" protocol (another variation on varint-prefixed protobuf messages over +an unstructured stream) and gRPC. It would be nice if we could get rid of one +of these to reduce (unneeded?) optionality. + +Since both the socket protocol and gRPC depend on protocol buffers, the +"socket" protocol is the most obvious choice to remove. While gRPC is more +complex, the set of languages that _have_ protobuf support but _lack_ gRPC +support is small. Moreover, gRPC is already widely used in the rest of the +ecosystem (including the Cosmos SDK). + +If some use case did arise later that can't work with gRPC, it would not be too +difficult for that application author to write a little proxy (in Go) that +bridges the convenient SDK APIs into a simpler protocol than gRPC. + +**Design principle:** It is better for an uncommon special case to carry the +burdens of its specialness, than to bake an escape hatch into the infrastructure. + +**Recommendation:** We should deprecate and remove the socket protocol. + +### Options for RPC Transport + +[ADR 057][adr-57] proposes using gRPC for the Tendermint RPC implementation. +This is still possible, but if we are able to simplify and decouple the +concerns as described above, I do not think it should be necessary. + +While JSON-RPC is not the best possible RPC protocol for all situations, it has +some advantages over gRPC for our domain. Specifically: + +- It is easy to call JSON-RPC manually from the command-line, which helps with + a common concern for the RPC service, local debugging and operations. + + Relatedly: JSON is relatively easy for humans to read and write, and it can + be easily copied and pasted to share sample queries and debugging results in + chat, issue comments, and so on. Ideally, the RPC service will not be used + for activities where the costs of a text protocol are important compared to + its legibility and manual usability benefits. + +- gRPC has an enormous dependency footprint for both clients and servers, and + many of the features it provides to support security and performance + (encryption, compression, streaming, etc.) are mostly irrelevant to local + use. Tendermint already needs to include a gRPC client for the remote signer, + but if we can avoid the need for a _client_ to depend on gRPC, that is a win + for usability. + +- If we intend to migrate light clients off RPC to use P2P entirely, there is + no advantage to forcing a temporary migration to gRPC along the way; and once + the light client is not dependent on the RPC service, the efficiency of the + protocol is much less important. + +- We can still get the benefits of generated data types using protocol buffers, even + without using gRPC: + + - Protobuf defines a standard JSON encoding for all message types so + languages with protobuf support do not need to worry about type mapping + oddities. + + - Using JSON means that even languages _without_ good protobuf support can + implement the protocol with a bit more work, and I expect this situation to + be rare. + +Even if a language lacks a good standard JSON-RPC mechanism, the protocol is +lightweight and can be implemented by simple send/receive over TCP or +Unix-domain sockets with no need for code generation, encryption, etc. gRPC +uses a complex HTTP/2 based transport that is not easily replicated. + +### Future Work + +The background and proposals sketched above focus on the existing structure of +Tendermint and improvements we can make in the short term. It is worthwhile to +also consider options for longer-term broader changes to the IPC ecosystem. +The following outlines some ideas at a high level: + +- **Consensus service:** Today, the application and the consensus node are + nominally connected only via ABCI. Tendermint was originally designed with + the assumption that all communication with the application should be mediated + by the consensus node. Based on further experience, however, the design goal + is now that the _application_ should be the mediator of application state. + + As noted above, however, ABCI is a client/server protocol, with the + application as the server. For outside clients that turns out to have been a + good choice, but it complicates the relationship between the application and + the consensus node: Previously transactions were entered via the node, now + they are entered via the app. + + We have worked around this by using the Tendermint RPC service to give the + application a "back channel" to the consensus node, so that it can push + transactions back into the consensus network. But the RPC service exposes a + lot of other functionality, too, including event subscription, block and + transaction queries, and a lot of node status information. + + Even if we can't easily "fix" the orientation of the ABCI relationship, we + could improve isolation by splitting out the parts of the RPC service that + the application needs as a back-channel, and sharing those _only_ with the + application. By defining a "consensus service", we could give the application + a way to talk back limited to only the capabilities it needs. This approach + has the benefit that we could do it without breaking existing use, and if we + later did "fix" the ABCI directionality, we could drop the special case + without disrupting the rest of the RPC interface. + +- **Event service:** Right now, the IBC relayer relies on the Tendermint RPC + service to provide a stream of block and transaction events, which it uses to + discover which transactions need relaying to other chains. While I think + that event subscription should eventually be handled via P2P, we could gain + some immediate benefit by splitting out event subscription from the rest of + the RPC service. + + In this model, an event subscription service would be exposed on the public + network, but on a different endpoint. This would remove the need for the RPC + service to support the websocket protocol, and would allow operators to + isolate potentially sensitive status query results from the public network. + + At the moment the relayers also use the RPC service to get block data for + synchronization, but work is already in progress to handle that concern via + the P2P layer. Once that's done, event subscription could be separated. + +Separating parts of the existing RPC service is not without cost: It might +require additional connection endpoints, for example, though it is also not too +difficult for multiple otherwise-independent services to share a connection. + +In return, though, it would become easier to reduce transport options and for +operators to independently control access to sensitive data. Considering the +viability and implications of these ideas is beyond the scope of this RFC, but +they are documented here since they follow from the background we have already +discussed. + +## References + +[abci]: https://github.com/tendermint/tendermint/tree/main/spec/abci +[rpc-service]: https://docs.tendermint.com/v0.34/rpc/ +[light-client]: https://docs.tendermint.com/v0.34/tendermint-core/light-client.html +[tm-cli]: https://github.com/tendermint/tendermint/tree/main/cmd/tendermint +[cosmos-sdk]: https://github.com/cosmos/cosmos-sdk/ +[local-client]: https://github.com/tendermint/tendermint/blob/main/abci/client/local_client.go +[socket-server]: https://github.com/tendermint/tendermint/blob/main/abci/server/socket_server.go +[sdk-grpc]: https://pkg.go.dev/github.com/cosmos/cosmos-sdk/types/tx#ServiceServer +[json-rpc]: https://www.jsonrpc.org/specification +[abci-conn]: https://github.com/tendermint/tendermint/blob/main/spec/abci/abci++_basic_concepts.md +[adr-57]: https://github.com/tendermint/tendermint/blob/main/docs/architecture/adr-057-RPC.md diff --git a/cometbft/v0.38/docs/rfc/tendermint-core/rfc-003-performance-questions.md b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-003-performance-questions.md new file mode 100644 index 00000000..fb4b582b --- /dev/null +++ b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-003-performance-questions.md @@ -0,0 +1,284 @@ +# RFC 003: Taxonomy of potential performance issues in Tendermint + +## Changelog + +- 2021-09-02: Created initial draft (@wbanfield) +- 2021-09-14: Add discussion of the event system (@wbanfield) + +## Abstract + +This document discusses the various sources of performance issues in Tendermint and +attempts to clarify what work may be required to understand and address them. + +## Background + +Performance, loosely defined as the ability of a software process to perform its work +quickly and efficiently under load and within reasonable resource limits, is a frequent +topic of discussion in the Tendermint project. +To effectively address any issues with Tendermint performance we need to +categorize the various issues, understand their potential sources, and gauge their +impact on users. + +Categorizing the different known performance issues will allow us to discuss and fix them +more systematically. This document proposes a rough taxonomy of performance issues +and highlights areas where more research into potential performance problems is required. + +Understanding Tendermint's performance limitations will also be critically important +as we make changes to many of its subsystems. Performance is a central concern for +upcoming decisions regarding the `p2p` protocol, RPC message encoding and structure, +database usage and selection, and consensus protocol updates. + + +## Discussion + +This section attempts to delineate the different sections of Tendermint functionality +that are often cited as having performance issues. It raises questions and suggests +lines of inquiry that may be valuable for better understanding Tendermint's performance issues. + +As a note: We should avoid quickly adding many microbenchmarks or package level benchmarks. +These are prone to being worse than useless as they can obscure what _should_ be +focused on: performance of the system from the perspective of a user. We should, +instead, tune performance with an eye towards user needs and actions users make. These users comprise +both operators of Tendermint chains and the people generating transactions for +Tendermint chains. Both of these sets of users are largely aligned in wanting an end-to-end +system that operates quickly and efficiently. + +REQUEST: The list below may be incomplete, if there are additional sections that are often +cited as creating poor performance, please comment so that they may be included. + +### P2P + +#### Claim: Tendermint cannot scale to large numbers of nodes + +A complaint has been reported that Tendermint networks cannot scale to large numbers of nodes. +The listed number of nodes a user reported as causing issue was in the thousands. +We don't currently have evidence about what the upper-limit of nodes that Tendermint's +P2P stack can scale to. + +We need to more concretely understand the source of issues and determine what layer +is causing a problem. It's possible that the P2P layer, in the absence of any reactors +sending data, is perfectly capable of managing thousands of peer connections. For +a reasonable networking and application setup, thousands of connections should not present any +issue for the application. + +We need more data to understand the problem directly. We want to drive the popularity +and adoption of Tendermint and this will mean allowing for chains with more validators. +We should follow up with users experiencing this issue. We may then want to add +a series of metrics to the P2P layer to better understand the inefficiencies it produces. + +The following metrics can help us understand the sources of latency in the Tendermint P2P stack: + +- Number of messages sent and received per second +- Time of a message spent on the P2P layer send and receive queues + +The following metrics exist and should be leveraged in addition to those added: + +- Number of peers node's connected to +- Number of bytes per channel sent and received from each peer + +### Sync + +#### Claim: Block Syncing is slow + +Bootstrapping a new node in a network to the height of the rest of the network is believed to +take longer than users would like. Block sync requires fetching all of the blocks from +peers and placing them into the local disk for storage. A useful line of inquiry +is understanding how quickly a perfectly tuned system _could_ fetch all of the state +over a network so that we understand how much overhead Tendermint actually adds. + +The operation is likely to be _incredibly_ dependent on the environment in which +the node is being run. The factors that will influence syncing include: + +1. Number of peers that a syncing node may fetch from. +2. Speed of the disk that a validator is writing to. +3. Speed of the network connection between the different peers that node is +syncing from. + +We should calculate how quickly this operation _could possibly_ complete for common chains and nodes. +To calculate how quickly this operation could possibly complete, we should assume that +a node is reading at line-rate of the NIC and writing at the full drive speed to its +local storage. Comparing this theoretical upper-limit to the actual sync times +observed by node operators will give us a good point of comparison for understanding +how much overhead Tendermint incurs. + +We should additionally add metrics to the blocksync operation to more clearly pinpoint +slow operations. The following metrics should be added to the block syncing operation: + +- Time to fetch and validate each block +- Time to execute a block +- Blocks sync'd per unit time + +### Application + +Applications performing complex state transitions have the potential to bottleneck +the Tendermint node. + +#### Claim: ABCI block delivery could cause slowdown + +ABCI delivers blocks in several methods: `BeginBlock`, `DeliverTx`, `EndBlock`, `Commit`. + +Tendermint delivers transactions one-by-one via the `DeliverTx` call. Most of the +transaction delivery in Tendermint occurs asynchronously and therefore appears unlikely to +form a bottleneck in ABCI. + +After delivering all transactions, Tendermint then calls the `Commit` ABCI method. +Tendermint [locks all access to the mempool][abci-commit-description] while `Commit` +proceeds. This means that an application that is slow to execute all of its +transactions or finalize state during the `Commit` method will prevent any new +transactions from being added to the mempool. Apps that are slow to commit will +prevent consensus from proceeded to the next consensus height since Tendermint +cannot validate block proposals or produce block proposals without the +AppHash obtained from the `Commit` method. We should add a metric for each +step in the ABCI protocol to track the amount of time that a node spends communicating +with the application at each step. + +#### Claim: ABCI serialization overhead causes slowdown + +The most common way to run a Tendermint application is using the Cosmos-SDK. +The Cosmos-SDK runs the ABCI application within the same process as Tendermint. +When an application is run in the same process as Tendermint, a serialization penalty +is not paid. This is because the local ABCI client does not serialize method calls +and instead passes the protobuf type through directly. This can be seen +in [local_client.go][abci-local-client-code]. + +Serialization and deserialization in the gRPC and socket protocol ABCI methods +may cause slowdown. While these may cause issue, they are not part of the primary +usecase of Tendermint and do not necessarily need to be addressed at this time. + +### RPC + +#### Claim: The Query API is slow + +The query API locks a mutex across the ABCI connections. This causes consensus to +slow during queries, as ABCI is no longer able to make progress. This is known +to be causing issue in the cosmos-sdk and is being addressed [in the sdk][sdk-query-fix] +but a more robust solution may be required. Adding metrics to each ABCI client connection +and message as described in the Application section of this document would allow us +to further introspect the issue here. + +#### Claim: RPC Serialization may cause slowdown + +The Tendermint RPC uses a modified version of JSON-RPC. This RPC powers the `broadcast_tx_*` methods, +which is a critical method for adding transactions to Tendermint at the moment. This method is +likely invoked quite frequently on popular networks. Being able to perform efficiently +on this common and critical operation is very important. The current JSON-RPC implementation +relies heavily on type introspection via reflection, which is known to be very slow in +Go. We should therefore produce benchmarks of this method to determine how much overhead +we are adding to what, is likely to be, a very common operation. + +The other JSON-RPC methods are much less critical to the core functionality of Tendermint. +While there may other points of performance consideration within the RPC, methods that do not +receive high volumes of requests should not be prioritized for performance consideration. + +NOTE: Previous discussion of the RPC framework was done in [ADR 57][adr-57] and +there is ongoing work to inspect and alter the JSON-RPC framework in [RFC 002][rfc-002]. +Much of these RPC-related performance considerations can either wait until the work of RFC 002 work is done or be +considered concordantly with the in-flight changes to the JSON-RPC. + +### Protocol + +#### Claim: Gossiping messages is a slow process + +Currently, for any validator to successfully vote in a consensus _step_, it must +receive votes from greater than 2/3 of the validators on the network. In many cases, +it's preferable to receive as many votes as possible from correct validators. + +This produces a quadratic increase in messages that are communicated as more validators join the network. +(Each of the N validators must communicate with all other N-1 validators). + +This large number of messages communicated per step has been identified to impact +performance of the protocol. Given that the number of messages communicated has been +identified as a bottleneck, it would be extremely valuable to gather data on how long +it takes for popular chains with many validators to gather all votes within a step. + +Metrics that would improve visibility into this include: + +- Amount of time for a node to gather votes in a step. +- Amount of time for a node to gather all block parts. +- Number of votes each node sends to gossip (i.e. not its own votes, but votes it is +transmitting for a peer). +- Total number of votes each node sends to receives (A node may receive duplicate votes +so understanding how frequently this occurs will be valuable in evaluating the performance +of the gossip system). + +#### Claim: Hashing Txs causes slowdown in Tendermint + +Using a faster hash algorithm for Tx hashes is currently a point of discussion +in Tendermint. Namely, it is being considered as part of the [modular hashing proposal][modular-hashing]. +It is currently unknown if hashing transactions in the Mempool forms a significant bottleneck. +Although it does not appear to be documented as slow, there are a few open github +issues that indicate a possible user preference for a faster hashing algorithm, +including [issue 2187][issue-2187] and [issue 2186][issue-2186]. + +It is likely worth investigating what order of magnitude Tx hashing takes in comparison to other +aspects of adding a Tx to the mempool. It is not currently clear if the rate of adding Tx +to the mempool is a source of user pain. We should not endeavor to make large changes to +consensus critical components without first being certain that the change is highly +valuable and impactful. + +### Digital Signatures + +#### Claim: Verification of digital signatures may cause slowdown in Tendermint + +Working with cryptographic signatures can be computationally expensive. The cosmos +hub uses [ed25519 signatures][hub-signature]. The library performing signature +verification in Tendermint on votes is [benchmarked][ed25519-bench] to be able to perform an `ed25519` +signature in 75μs on a decently fast CPU. A validator in the Cosmos Hub performs +3 sets of verifications on the signatures of the 140 validators in the Hub +in a consensus round, during block verification, when verifying the prevotes, and +when verifying the precommits. With no batching, this would be roughly `3ms` per +round. It is quite unlikely, therefore, that this accounts for any serious amount +of the ~7 seconds of block time per height in the Hub. + +This may cause slowdown when syncing, since the process needs to constantly verify +signatures. It's possible that improved signature aggregation will lead to improved +light client or other syncing performance. In general, a metric should be added +to track block rate while blocksyncing. + +#### Claim: Our use of digital signatures in the consensus protocol contributes to performance issue + +Currently, Tendermint's digital signature verification requires that all validators +receive all vote messages. Each validator must receive the complete digital signature +along with the vote message that it corresponds to. This means that all N validators +must receive messages from at least 2/3 of the N validators in each consensus +round. Given the potential for oddly shaped network topologies and the expected +variable network roundtrip times of a few hundred milliseconds in a blockchain, +it is highly likely that this amount of gossiping is leading to a significant amount +of the slowdown in the Cosmos Hub and in Tendermint consensus. + +### Tendermint Event System + +#### Claim: The event system is a bottleneck in Tendermint + +The Tendermint Event system is used to communicate and store information about +internal Tendermint execution. The system uses channels internally to send messages +to different subscribers. Sending an event [blocks on the internal channel][event-send]. +The default configuration is to [use an unbuffered channel for event publishes][event-buffer-capacity]. +Several consumers of the event system also use an unbuffered channel for reads. +An example of this is the [event indexer][event-indexer-unbuffered], which takes an +unbuffered subscription to the event system. The result is that these unbuffered readers +can cause writes to the event system to block or slow down depending on contention in the +event system. This has implications for the consensus system, which [publishes events][consensus-event-send]. +To better understand the performance of the event system, we should add metrics to track the timing of +event sends. The following metrics would be a good start for tracking this performance: + +- Time in event send, labeled by Event Type +- Time in event receive, labeled by subscriber +- Event throughput, measured in events per unit time. + +### References + +[modular-hashing]: https://github.com/tendermint/tendermint/pull/6773 +[issue-2186]: https://github.com/tendermint/tendermint/issues/2186 +[issue-2187]: https://github.com/tendermint/tendermint/issues/2187 +[rfc-002]: https://github.com/tendermint/tendermint/pull/6913 +[adr-57]: https://github.com/tendermint/tendermint/blob/main/docs/architecture/adr-057-RPC.md +[abci-commit-description]: https://github.com/tendermint/tendermint/blob/main/spec/abci/abci++_methods.md#commit +[abci-local-client-code]: https://github.com/tendermint/tendermint/blob/511bd3eb7f037855a793a27ff4c53c12f085b570/abci/client/local_client.go#L84 +[hub-signature]: https://github.com/cosmos/gaia/blob/0ecb6ed8a244d835807f1ced49217d54a9ca2070/docs/resources/genesis.md#consensus-parameters +[ed25519-bench]: https://github.com/oasisprotocol/curve25519-voi/blob/d2e7fc59fe38c18ca990c84c4186cba2cc45b1f9/PERFORMANCE.md +[event-send]: https://github.com/tendermint/tendermint/blob/5bd3b286a2b715737f6d6c33051b69061d38f8ef/libs/pubsub/pubsub.go#L338 +[event-buffer-capacity]: https://github.com/tendermint/tendermint/blob/5bd3b286a2b715737f6d6c33051b69061d38f8ef/types/event_bus.go#L14 +[event-indexer-unbuffered]: https://github.com/tendermint/tendermint/blob/5bd3b286a2b715737f6d6c33051b69061d38f8ef/state/indexer/indexer_service.go#L39 +[consensus-event-send]: https://github.com/tendermint/tendermint/blob/5bd3b286a2b715737f6d6c33051b69061d38f8ef/internal/consensus/state.go#L1573 +[sdk-query-fix]: https://github.com/cosmos/cosmos-sdk/pull/10045 diff --git a/cometbft/v0.38/docs/rfc/tendermint-core/rfc-004-e2e-framework.rst b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-004-e2e-framework.rst new file mode 100644 index 00000000..7c7b96fc --- /dev/null +++ b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-004-e2e-framework.rst @@ -0,0 +1,213 @@ +======================================== +RFC 004: E2E Test Framework Enhancements +======================================== + +Changelog +--------- + +- 2021-09-14: started initial draft (@tychoish) + +Abstract +-------- + +This document discusses a series of improvements to the e2e test framework +that we can consider during the next few releases to help boost confidence in +Tendermint releases, and improve developer efficiency. + +Background +---------- + +During the 0.35 release cycle, the E2E tests were a source of great +value, helping to identify a number of bugs before release. At the same time, +the tests were not consistently passing during this time, thereby reducing +their value, and forcing the core development team to allocate time and energy +to maintaining and chasing down issues with the e2e tests and the test +harness. The experience of this release cycle calls to mind a series of +improvements to the test framework, and this document attempts to capture +these improvements, along with motivations, and potential for impact. + +Projects +-------- + +Flexible Workload Generation +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Presently the e2e suite contains a single workload generation pattern, which +exists simply to ensure that the test networks have some work during their +runs. However, the shape and volume of the work is very consistent and is very +gentle to help ensure test reliability. + +We don't need a complex workload generation framework, but being able to have +a few different workload shapes available for test networks, both generated and +hand-crafted, would be useful. + +Workload patterns/configurations might include: + +- transaction targeting patterns (include light nodes, round robin, target + individual nodes) + +- variable transaction size over time. + +- transaction broadcast option (synchronously, checked, fire-and-forget, + mixed). + +- number of transactions to submit. + +- non-transaction workloads: (evidence submission, query, event subscription.) + +Configurable Generator +~~~~~~~~~~~~~~~~~~~~~~ + +The nightly e2e suite is defined by the `testnet generator +`_, +and it's difficult to add dimensions or change the focus of the test suite in +any way without modifying the implementation of the generator. If the +generator were more configurable, potentially via a file rather than in +the Go implementation, we could modify the focus of the test suite on the +fly. + +Features that we might want to configure: + +- number of test networks to generate of various topologies, to improve + coverage of different configurations. + +- test application configurations (to modify the latency of ABCI calls, etc.) + +- size of test networks. + +- workload shape and behavior. + +- initial sync and catch-up configurations. + +The workload generator currently provides runtime options for limiting the +generator to specific types of P2P stacks, and for generating multiple groups +of test cases to support parallelism. The goal is to extend this pattern and +avoid hardcoding the matrix of test cases in the generator code. Once the +testnet configuration generation behavior is configurable at runtime, +developers may be able to use the e2e framework to validate changes before +landing changes that break e2e tests a day later. + +In addition to the autogenerated suite, it might make sense to maintain a +small collection of hand-crafted cases that exercise configurations of +concern, to run as part of the nightly (or less frequent) loop. + +Implementation Plan Structure +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +As a development team, we should determine the features should impact the e2e +testing early in the development cycle, and if we intend to modify the e2e +tests to exercise a feature, we should identify this early and begin the +integration process as early as possible. + +To facilitate this, we should adopt a practice whereby we exercise specific +features that are currently under development more rigorously in the e2e +suite, and then as development stabilizes we can reduce the number or weight +of these features in the suite. + +As of 0.35 there are essentially two end to end tests: the suite of 64 +generated test networks, and the hand crafted `ci.toml` test case. The +generated test cases help provide systemtic coverage, while the `ci` run +provides coverage for a large number of features. + +Reduce Cycle Time +~~~~~~~~~~~~~~~~~ + +One of the barriers to leveraging the e2e framework, and one of the challenges +in debugging failures, is the cycle time of running a single test iteration is +quite high: 5 minutes to build the docker image, plus the time to run the test +or tests. + +There are a number of improvements and enhancements that can reduce the cycle +time in practice: + +- reduce the amount of time required to build the docker image used in these + tests. Without the dependency on CGo, the tendermint binaries could be + (cross) compiled outside of the docker container and then injected into + them, which would take better advantage of docker's native caching, + although, without the dependency on CGo there would be no hard requirement + for the e2e tests to use docker. + +- support test parallelism. Because of the way the testnets are orchestrated + a single system can really only run one network at a time. For executions + (local or remote) with more resources, there's no reason to run a few + networks in parallel to reduce the feedback time. + +- prune testnet configurations that are unlikely to provide good signal, to + shorten the time to feedback. + +- apply some kind of tiered approach to test execution, to improve the + legibility of the test result. For example order tests by the dependency of + their features, or run test networks without perturbations before running + that configuration with perturbations, to be able to isolate the impact of + specific features. + +- orchestrate the test harness directly from go test rather than via a special + harness and shell scripts so e2e tests may more naively fit into developers + existing workflows. + +Many of these improvements, particularly, reducing the build time will also +reduce the time to get feedback during automated builds. + +Deeper Insights +~~~~~~~~~~~~~~~ + +When a test network fails, it's incredibly difficult to understand _why_ the +network failed, as the current system provides very little insight into the +system outside of the process logs. When a test network stalls or fails +developers should be able to quickly and easily get a sense of the state of +the network and all nodes. + +Improvements in persuit of this goal, include functionality that would help +node operators in production environments by improving the quality and utility +of the logging messages and other reported metrics, but also provide some +tools to collect and aggregate this data for developers in the context of test +networks. + +- Interleave messages from all nodes in the network to be able to correlate + events during the test run. + +- Collect structured metrics of the system operation (CPU/MEM/IO) during the + test run, as well as from each tendermint/application process. + +- Build (simple) tools to be able to render and summarize the data collected + during the test run to answer basic questions about test outcome. + +Flexible Assertions +~~~~~~~~~~~~~~~~~~~ + +Currently, all assertions run for every test network, which makes the +assertions pretty bland, and the framework primarily useful as a smoke-test +framework, but it might be useful to be able to write and run different +tests for different configurations. This could allow us to test outside of the +happy-path. + +In general our existing assertions occupy a fraction of the total test time, +so the relative cost of adding a few extra test assertions would be of limited +cost, and could help build confidence. + +Additional Kinds of Testing +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The existing e2e suite, exercises networks of nodes that have homogeneous +tendermint version, stable configuration, that are expected to make +progress. There are many other possible test configurations that may be +interesting to engage with. These could include dimensions, such as: + +- Multi-version testing to exercise our compatibility guarantees for networks + that might have different tendermint versions. + +- As a flavor or mult-version testing, include upgrade testing, to build + confidence in migration code and procedures. + +- Additional test applications, particularly practical-type applciations + including some that use gaiad and/or the cosmos-sdk. Test-only applications + that simulate other kinds of applications (e.g. variable application + operation latency.) + +- Tests of "non-viable" configurations that ensure that forbidden combinations + lead to halts. + +References +---------- + +- `ADR 66: End-to-End Testing <../architecture/adr-066-e2e-testing.md>`_ diff --git a/cometbft/v0.38/docs/rfc/tendermint-core/rfc-005-event-system.rst b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-005-event-system.rst new file mode 100644 index 00000000..b70ee6c0 --- /dev/null +++ b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-005-event-system.rst @@ -0,0 +1,122 @@ +===================== +RFC 005: Event System +===================== + +Changelog +--------- + +- 2021-09-17: Initial Draft (@tychoish) + +Abstract +-------- + +The event system within Tendermint, which supports a lot of core +functionality, also represents a major infrastructural liability. As part of +our upcoming review of the RPC interfaces and our ongoing thoughts about +stability and performance, as well as the preparation for Tendermint 1.0, we +should revisit the design and implementation of the event system. This +document discusses both the current state of the system and potential +directions for future improvement. + +Background +---------- + +Current State of Events +~~~~~~~~~~~~~~~~~~~~~~~ + +The event system makes it possible for clients, both internal and external, +to receive notifications of state replication events, such as new blocks, +new transactions, validator set changes, as well as intermediate events during +consensus. Because the event system is very cross cutting, the behavior and +performance of the event publication and subscription system has huge impacts +for all of Tendermint. + +The subscription service is exposed over the RPC interface, but also powers +the indexing (e.g. to an external database,) and is the mechanism by which +`BroadcastTxCommit` is able to wait for transactions to land in a block. + +The current pubsub mechanism relies on a couple of buffered channels, +primarily between all event creators and subscribers, but also for each +subscription. The result of this design is that, in some situations with the +right collection of slow subscription consumers the event system can put +backpressure on the consensus state machine and message gossiping in the +network, thereby causing nodes to lag. + +Improvements +~~~~~~~~~~~~ + +The current system relies on implicit, bounded queues built by the buffered channels, +and though threadsafe, can force all activity within Tendermint to serialize, +which does not need to happen. Additionally, timeouts for subscription +consumers related to the implementation of the RPC layer, may complicate the +use of the system. + +References +~~~~~~~~~~ + +- Legacy Implementation + - `publication of events `_ + - `send operation `_ + - `send loop `_ +- Related RFCs + - `RFC 002: IPC Ecosystem <./rfc-002-ipc-ecosystem.md>`_ + - `RFC 003: Performance Questions <./rfc-003-performance-questions.md>`_ + +Discussion +---------- + +Changes to Published Events +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +As part of this process, the Tendermint team should do a study of the existing +event types and ensure that there are viable production use cases for +subscriptions to all event types. Instinctively it seems plausible that some +of the events may not be useable outside of tendermint, (e.g. ``TimeoutWait`` +or ``NewRoundStep``) and it might make sense to remove them. Certainly, it +would be good to make sure that we don't maintain infrastructure for unused or +un-useful message indefinitely. + +Blocking Subscription +~~~~~~~~~~~~~~~~~~~~~ + +The blocking subscription mechanism makes it possible to have *send* +operations into the subscription channel be un-buffered (the event processing +channel is still buffered.) In the blocking case, events from one subscription +can block processing that event for other non-blocking subscriptions. The main +case, it seems for blocking subscriptions is ensuring that a transaction has +been committed to a block for ``BroadcastTxCommit``. Removing blocking +subscriptions entirely, and potentially finding another way to implement +``BroadcastTxCommit``, could lead to important simplifications and +improvements to throughput without requiring large changes. + +Subscription Identification +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Before `#6386 `_, all +subscriptions were identified by the combination of a client ID and a query, +and with that change, it became possible to identify all subscription given +only an ID, but compatibility with the legacy identification means that there's a +good deal of legacy code as well as client side efficiency that could be +improved. + +Pubsub Changes +~~~~~~~~~~~~~~ + +The pubsub core should be implemented in a way that removes the possibility of +backpressure from the event system to impact the core system *or* for one +subscription to impact the behavior of another area of the +system. Additionally, because the current system is implemented entirely in +terms of a collection of buffered channels, the event system (and large +numbers of subscriptions) can be a source of memory pressure. + +These changes could include: + +- explicit cancellation and timeouts promulgated from callers (e.g. RPC end + points, etc,) this should be done using contexts. + +- subscription system should be able to spill to disk to avoid putting memory + pressure on the core behavior of the node (consensus, gossip). + +- subscriptions implemented as cursors rather than channels, with either + condition variables to simulate the existing "push" API or a client side + iterator API with some kind of long polling-type interface. diff --git a/cometbft/v0.38/docs/rfc/tendermint-core/rfc-006-event-subscription.md b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-006-event-subscription.md new file mode 100644 index 00000000..0e03c119 --- /dev/null +++ b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-006-event-subscription.md @@ -0,0 +1,204 @@ +# RFC 006: Event Subscription + +## Changelog + +- 30-Oct-2021: Initial draft (@creachadair) + +## Abstract + +The Tendermint consensus node allows clients to subscribe to its event stream +via methods on its RPC service. The ability to view the event stream is +valuable for clients, but the current implementation has some deficiencies that +make it difficult for some clients to use effectively. This RFC documents these +issues and discusses possible approaches to solving them. + + +## Background + +A running Tendermint consensus node exports a [JSON-RPC service][rpc-service] +that provides a [large set of methods][rpc-methods] for inspecting and +interacting with the node. One important cluster of these methods are the +`subscribe`, `unsubscribe`, and `unsubscribe_all` methods, which permit clients +to subscribe to a filtered stream of the [events generated by the node][events] +as it runs. + +Unlike the other methods of the service, the methods in the "event +subscription" cluster are not accessible via [ordinary HTTP GET or POST +requests][rpc-transport], but require upgrading the HTTP connection to a +[websocket][ws]. This is necessary because the `subscribe` request needs a +persistent channel to deliver results back to the client, and an ordinary HTTP +connection does not reliably persist across multiple requests. Since these +methods do not work properly without a persistent channel, they are _only_ +exported via a websocket connection, and are not routed for plain HTTP. + + +## Discussion + +There are some operational problems with the current implementation of event +subscription in the RPC service: + +- **Event delivery is not valid JSON-RPC.** When a client issues a `subscribe` + request, the server replies (correctly) with an initial empty acknowledgement + (`{}`). After that, each matching event is delivered "unsolicited" (without + another request from the client), as a separate [response object][json-response] + with the same ID as the initial request. + + This matters because it means a standard JSON-RPC client library can't + interact correctly with the event subscription mechanism. + + Even for clients that can handle unsolicited values pushed by the server, + these responses are invalid: They have an ID, so they cannot be treated as + [notifications][json-notify]; but the ID corresponds to a request that was + already completed. In practice, this means that general-purpose JSON-RPC + libraries cannot use this method correctly -- it requires a custom client. + + The Go RPC client from the Tendermint core can support this case, but clients + in other languages have no easy solution. + + This is the cause of issue [#2949][issue2949]. + +- **Subscriptions are terminated by disconnection.** When the connection to the + client is interrupted, the subscription is silently dropped. + + This is a reasonable behavior, but it matters because a client whose + subscription is dropped gets no useful error feedback, just a closed + connection. Should they try again? Is the node overloaded? Was the client + too slow? Did the caller forget to respond to pings? Debugging these kinds + of failures is unnecessarily painful. + + Websockets compound this, because websocket connections time out if no + traffic is seen for a while, and keeping them alive requires active + cooperation between the client and server. With a plain TCP socket, liveness + is handled transparently by the keepalive mechanism. On a websocket, + however, one side has to occasionally send a PING (if the connection is + otherwise idle). The other side must return a matching PONG in time, or the + connection is dropped. Apart from being tedious, this is highly susceptible + to CPU load. + + The Tendermint Go implementation automatically sends and responds to pings. + Clients in other languages (or not wanting to use the Tendermint libraries) + need to handle it explicitly. This burdens the client for no practical + benefit: A subscriber has no information about when matching events may be + available, so it shouldn't have to participate in keeping the connection + alive. + +- **Mismatched load profiles.** Most of the RPC service is mainly important for + low-volume local use, either by the application the node serves (e.g., the + ABCI methods) or by the node operator (e.g., the info methods). Event + subscription is important for remote clients, and may represent a much higher + volume of traffic. + + This matters because both are using the same JSON-RPC mechanism. For + low-volume local use, the ergonomics of JSON-RPC are a good fit: It's easy to + issue queries from the command line (e.g., using `curl`) or to write scripts + that call the RPC methods to monitor the running node. + + For high-volume remote use, JSON-RPC is not such a good fit: Even leaving + aside the non-standard delivery protocol mentioned above, the time and memory + cost of encoding event data matters for the stability of the node when there + can be potentially hundreds of subscribers. Moreover, a subscription is + long-lived compared to most RPC methods, in that it may persist as long the + node is active. + +- **Mismatched security profiles.** The RPC service exports several methods + that should not be open to arbitrary remote callers, both for correctness + reasons (e.g., `remove_tx` and `broadcast_tx_*`) and for operational + stability reasons (e.g., `tx_search`). A node may still need to expose + events, however, to support UI tools. + + This matters, because all the methods share the same network endpoint. While + it is possible to block the top-level GET and POST handlers with a proxy, + exposing the `/websocket` handler exposes not _only_ the event subscription + methods, but the rest of the service as well. + +### Possible Improvements + +There are several things we could do to improve the experience of developers +who need to subscribe to events from the consensus node. These are not all +mutually exclusive. + +1. **Split event subscription into a separate service**. Instead of exposing + event subscription on the same endpoint as the rest of the RPC service, + dedicate a separate endpoint on the node for _only_ event subscription. The + rest of the RPC services (_sans_ events) would remain as-is. + + This would make it easy to disable or firewall outside access to sensitive + RPC methods, without blocking access to event subscription (and vice versa). + This is probably worth doing, even if we don't take any of the other steps + described here. + +2. **Use a different protocol for event subscription.** There are various ways + we could approach this, depending how much we're willing to shake up the + current API. Here are sketches of a few options: + + - Keep the websocket, but rework the API to be more JSON-RPC compliant, + perhaps by converting event delivery into notifications. This is less + up-front change for existing clients, but retains all of the existing + implementation complexity, and doesn't contribute much toward more serious + performance and UX improvements later. + + - Switch from websocket to plain HTTP, and rework the subscription API to + use a more conventional request/response pattern instead of streaming. + This is a little more up-front work for existing clients, but leverages + better library support for clients not written in Go. + + The protocol would become more chatty, but we could mitigate that with + batching, and in return we would get more control over what to do about + slow clients: Instead of simply silently dropping them, as we do now, we + could drop messages and signal the client that they missed some data ("M + dropped messages since your last poll"). + + This option is probably the best balance between work, API change, and + benefit, and has a nice incidental effect that it would be easier to debug + subscriptions from the command-line, like the other RPC methods. + + - Switch to gRPC: Preserves a persistent connection and gives us a more + efficient binary wire format (protobuf), at the cost of much more work for + clients and harder debugging. This may be the best option if performance + and server load are our top concerns. + + Given that we are currently using JSON-RPC, however, I'm not convinced the + costs of encoding and sending messages on the event subscription channel + are the limiting factor on subscription efficiency, however. + +3. **Delegate event subscriptions to a proxy.** Give responsibility for + managing event subscription to a proxy that runs separately from the node, + and switch the node to push events to the proxy (like a webhook) instead of + serving subscribers directly. This is more work for the operator (another + process to configure and run) but may scale better for big networks. + + I mention this option for completeness, but making this change would be a + fairly substantial project. If we want to consider shifting responsibility + for event subscription outside the node anyway, we should probably be more + systematic about it. For a more principled approach, see point (4) below. + +4. **Move event subscription downstream of indexing.** We are already planning + to give applications more control over event indexing. By extension, we + might allow the application to also control how events are filtered, + queried, and subscribed. Having the application control these concerns, + rather than the node, might make life easier for developers building UI and + tools for that application. + + This is a much larger change, so I don't think it is likely to be practical + in the near-term, but it's worth considering as a broader option. Some of + the existing code for filtering and selection could be made more reusable, + so applications would not need to reinvent everything. + + +## References + +- [Tendermint RPC service][rpc-service] +- [Tendermint RPC routes][rpc-methods] +- [Discussion of the event system][events] +- [Discussion about RPC transport options][rpc-transport] (from RFC 002) +- [RFC 6455: The websocket protocol][ws] +- [JSON-RPC 2.0 Specification](https://www.jsonrpc.org/specification) + +[rpc-service]: https://docs.tendermint.com/v0.34/rpc/ +[rpc-methods]: https://github.com/tendermint/tendermint/blob/main/rpc/core/routes.go#L12 +[events]: ./rfc-005-event-system.rst +[rpc-transport]: ./rfc-002-ipc-ecosystem.md#rpc-transport +[ws]: https://datatracker.ietf.org/doc/html/rfc6455 +[json-response]: https://www.jsonrpc.org/specification#response_object +[json-notify]: https://www.jsonrpc.org/specification#notification +[issue2949]: https://github.com/tendermint/tendermint/issues/2949 diff --git a/cometbft/v0.38/docs/rfc/tendermint-core/rfc-007-deterministic-proto-bytes.md b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-007-deterministic-proto-bytes.md new file mode 100644 index 00000000..c1521753 --- /dev/null +++ b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-007-deterministic-proto-bytes.md @@ -0,0 +1,138 @@ +# RFC 007 : Deterministic Proto Byte Serialization + +## Changelog + +- 09-Dec-2021: Initial draft (@williambanfield). + +## Abstract + +This document discusses the issue of stable byte-representation of serialized messages +within Tendermint and describes a few possible routes that could be taken to address it. + +## Background + +We use the byte representations of wire-format proto messages to produce +and verify hashes of data within the Tendermint codebase as well as for +producing and verifying cryptographic signatures over these signed bytes. + +The protocol buffer [encoding spec][proto-spec-encoding] does not guarantee that the byte representation +of a protocol buffer message will be the same between two calls to an encoder. +While there is a mode to force the encoder to produce the same byte representation +of messages within a single binary, these guarantees are not good enough for our +use case in Tendermint. We require multiple different versions of a binary running +Tendermint to be able to inter-operate. Additionally, we require that multiple different +systems written in _different languages_ be able to participate in different aspects +of the protocols of Tendermint and be able to verify the integrity of the messages +they each produce. + +While this has not yet created a problem that we know of in a running network, we should +make sure to provide stronger guarantees around the serialized representation of the messages +used within the Tendermint consensus algorithm to prevent any issue from occurring. + + +## Discussion + +Proto has the following points of variability that can produce non-deterministic byte representation: + +1. Encoding order of fields within a message. + +Proto allows fields to be encoded in any order and even be repeated. + +2. Encoding order of elements of a repeated field. + +`repeated` fields in a proto message can be serialized in any order. + +3. Presence or absence of default values. + +Types in proto have defined default values similar to Go's zero values. +Writing or omitting a default value are both legal ways of encoding a wire message. + +4. Serialization of 'unknown' fields. + +Unknown fields can be present when a message is created by a binary with a newer +version of the proto that contains fields that the deserializer in a different +binary does not yet know about. Deserializers in binaries that do not know about the field +will maintain the bytes of the unknown field but not place them into the deserialized structure. + +We have a few options to consider when producing this stable representation. + +### Options for deterministic byte representation + +#### Use only compliant serializers and constrain field usage + +According to [Cosmos-SDK ADR-27][cosmos-sdk-adr-27], when message types obey a simple +set of rules, gogoproto produces a consistent byte representation of serialized messages. +This seems promising, although more research is needed to guarantee gogoproto always +produces a consistent set of bytes on serialized messages. This would solve the problem +within Tendermint as written in Go, but would require ensuring that there are similar +serializers written in other languages that produce the same output as gogoproto. + +#### Reorder serialized bytes to ensure determinism + +The serialized form of a proto message can be transformed into a canonical representation +by applying simple rules to the serialized bytes. Re-ordering the serialized bytes +would allow Tendermint to produce a canonical byte representation without having to +simultaneously maintain a custom proto marshaller. + +This could be implemented as a function in many languages that performed the following +producing bytes to sign or hashing: + +1. Does not add any of the data from unknown fields into the type to hash. + +Tendermint should not run into a case where it needs to verify the integrity of +data with unknown fields for the following reasons: + +The purpose of checking hash equality within Tendermint is to ensure that +its local copy of data matches the data that the network agreed on. There should +therefore not be a case where a process is checking hash equality using data that it did not expect +to receive. What the data represent may be opaque to the process, such as when checking the +transactions in a block, _but the process will still have expected to receive this data_, +despite not understanding what their internal structure is. It's not clear what it would +mean to verify that a block contains data that a process does not know about. + +The same reasoning applies for signature verification within Tendermint. Processes +verify that a digital signature signed over a set of bytes by locally reconstructing the +data structure that the digital signature signed using the process's local data. + +2. Reordered all message fields to be in tag-sorted order. + +Tag-sorting top-level fields will place all fields of the same tag in a adjacent +to eachother within the serialized representation. + +3. Reordered the contents of all `repeated` fields to be in lexicographically sorted order. + +`repeated` fields will appear in a message as having the same tag but will contain different +contents. Therefore, lexicographical sorting will produce a stable ordering of +fields with the same tag. + +4. Deleted all default values from the byte representation. + +Encoders can include default values or omit them. Most encoders appear to omit them +but we may wish to delete them just to be safe. + +5. Recursively performed these operations on any length-delimited subfields. + +Length delimited fields may contain messages, strings, or just bytes. However, +it's not possible to know what data is being represented by such a field. +A 'string' may happen to have the same structure as an embedded message and we cannot +disambiguate. For this reason, we must apply these same rules to all subfields that +may contain messages. Because we cannot know if we have totally mangled the interior 'string' +or not, this data should never be deserialized or used for anything beyond hashing. + +A **prototype** implementation by @creachadair of this can be found in [the wirepb repo][wire-pb]. +This could be implemented in multiple languages more simply than ensuring that there are +canonical proto serializers that match in each language. + +### Future work + +We should add clear documentation to the Tendermint codebase every time we +compare hashes of proto messages or use proto serialized bytes to produces a +digital signatures that we have been careful to ensure that the hashes are performed +properly. + +### References + +[proto-spec-encoding]: https://developers.google.com/protocol-buffers/docs/encoding +[cosmos-sdk-adr-27]: https://github.com/cosmos/cosmos-sdk/blob/master/docs/architecture/adr-027-deterministic-protobuf-serialization.md +[wire-pb]: https://github.com/creachadair/wirepb + diff --git a/cometbft/v0.38/docs/rfc/tendermint-core/rfc-008-do-not-panic.md b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-008-do-not-panic.md new file mode 100644 index 00000000..ec8c08f5 --- /dev/null +++ b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-008-do-not-panic.md @@ -0,0 +1,139 @@ +# RFC 008: Don't Panic + +## Changelog + +- 2021-12-17: initial draft (@tychoish) + +## Abstract + +Today, the Tendermint core codebase has panics in a number of cases as +a response to exceptional situations. These panics complicate testing, +and might make tendermint components difficult to use as a library in +some circumstances. This document outlines a project of converting +panics to errors and describes the situations where its safe to +panic. + +## Background + +Panics in Go are a great mechanism for aborting the current execution +for truly exceptional situations (e.g. memory errors, data corruption, +processes initialization); however, because they resemble exceptions +in other languages, it can be easy to over use them in the +implementation of software architectures. This certainly happened in +the history of Tendermint, and as we embark on the project of +stabilizing the package, we find ourselves in the right moment to +reexamine our use of panics, and largely where panics happen in the +code base. + +There are still some situations where panics are acceptable and +desireable, but it's important that Tendermint, as a project, comes to +consensus--perhaps in the text of this document--on the situations +where it is acceptable to panic. + +### References + +- [Defer Panic and Recover](https://go.dev/blog/defer-panic-and-recover) +- [Why Go gets exceptions right](https://dave.cheney.net/tag/panic) +- [Don't panic](https://dave.cheney.net/practical-go/presentations/gophercon-singapore-2019.html#_dont_panic) + +## Discussion + +### Acceptable Panics + +#### Initialization + +It is unambiguously safe (and desireable) to panic in `init()` +functions in response to any kind of error. These errors are caught by +tests, and occur early enough in process initialization that they +won't cause unexpected runtime crashes. + +Other code that is called early in process initialization MAY panic, +in some situations if it's not possible to return an error or cause +the process to abort early, although these situations should be +vanishingly slim. + +#### Data Corruption + +If Tendermint code encounters an inconsistency that could be +attributed to data corruption or a logical impossibility it is safer +to panic and crash the process than continue to attempt to make +progress in these situations. + +Examples including reading data out of the storage engine that +is invalid or corrupt, or encountering an ambiguous situation where +the process should halt. Generally these forms of corruption are +detected after interacting with a trusted but external data source, +and reflect situations where the author thinks its safer to terminate +the process immediately rather than allow execution to continue. + +#### Unrecoverable Consensus Failure + +In general, a panic should be used in the case of unrecoverable +consensus failures. If a process detects that the network is +behaving in an incoherent way and it does not have a clearly defined +and mechanism for recovering, the process should panic. + +#### Static Validity + +It is acceptable to panic for invariant violations, within a library +or package, in situations that should be statically impossible, +because there is no way to make these kinds of assertions at compile +time. + +For example, type-asserting `interface{}` values returned by +`container/list` and `container/heap` (and similar), is acceptable, +because package authors should have exclusive control of the inputs to +these containers. Packages should not expose the ability to add +arbitrary values to these data structures. + +#### Controlled Panics Within Libraries + +In some algorithms with highly recursive structures or very nested +call patterns, using a panic, in combination with conditional recovery +handlers results in more manageable code. Ultimately this is a limited +application, and implementations that use panics internally should +only recover conditionally, filtering out panics rather than ignoring +or handling all panics. + +#### Request Handling + +Code that handles responses to incoming/external requests +(e.g. `http.Handler`) should avoid panics, but practice this isn't +totally possible, and it makes sense that request handlers have some +kind of default recovery mechanism that will prevent one request from +terminating a service. + +### Unacceptable Panics + +In **no** other situation is it acceptable for the code to panic: + +- there should be **no** controlled panics that callers are required + to handle across library/package boundaries. +- callers of library functions should not expect panics. +- ensuring that arbitrary go routines can't panic. +- ensuring that there are no arbitrary panics in core production code, + espically code that can run at any time during the lifetime of a + process. +- all test code and fixture should report normal test assertions with + a mechanism like testify's `require` assertion rather than calling + panic directly. + +The goal of this increased "panic rigor" is to ensure that any escaped +panic is reflects a fixable bug in Tendermint. + +### Removing Panics + +The process for removing panics involve a few steps, and will be part +of an ongoing process of code modernization: + +- converting existing explicit panics to errors in cases where it's + possible to return an error, the errors can and should be handled, and returning + an error would not lead to data corruption or cover up data + corruption. + +- increase rigor around operations that can cause runtime errors, like + type assertions, nil pointer errors, array bounds access issues, and + either avoid these situations or return errors where possible. + +- remove generic panic handlers which could cover and hide known + panics. diff --git a/cometbft/v0.38/docs/rfc/tendermint-core/rfc-009-consensus-parameter-upgrades.md b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-009-consensus-parameter-upgrades.md new file mode 100644 index 00000000..e89c4c1a --- /dev/null +++ b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-009-consensus-parameter-upgrades.md @@ -0,0 +1,128 @@ +# RFC 009 : Consensus Parameter Upgrade Considerations + +## Changelog + +- 06-Jan-2011: Initial draft (@williambanfield). + +## Abstract + +This document discusses the challenges of adding additional consensus parameters +to Tendermint and proposes a few solutions that can enable addition of consensus +parameters in a backwards-compatible way. + +## Background + +This section provides an overview of the issues of adding consensus parameters +to Tendermint. + +### Hash Compatibility + +Tendermint produces a hash of a subset of the consensus parameters. The values +that are hashed currently are the `BlockMaxGas` and the `BlockMaxSize`. These +are currently in the [HashedParams struct][hashed-params]. This hash is included +in the block and validators use it to validate that their local view of the consensus +parameters matches what the rest of the network is configured with. + +Any new consensus parameters added to Tendermint should be included in this +hash. This presents a challenge for verification of historical blocks when consensus +parameters are added. If a network produced blocks with a version of Tendermint that +did not yet have the new consensus parameters, the parameter hash it produced will +not reference the new parameters. Any nodes joining the network with the newer +version of Tendermint will have the new consensus parameters. Tendermint will need +to handle this case so that new versions of Tendermint with new consensus parameters +can still validate old blocks correctly without having to do anything overly complex +or hacky. + +### Allowing Developer-Defined Values and the `EndBlock` Problem + +When new consensus parameters are added, application developers may wish to set +values for them so that the developer-defined values may be used as soon as the +software upgrades. We do not currently have a clean mechanism for handling this. + +Consensus parameter updates are communicated from the application to Tendermint +within `EndBlock` of some height `H` and take effect at the next height, `H+1`. +This means that for updates that add a consensus parameter, there is a single +height where the new parameters cannot take effect. The parameters did not exist +in the version of the software that emitted the `EndBlock` response for height `H-1`, +so they cannot take effect at height `H`. The first height that the updated params +can take effect is height `H+1`. As of now, height `H` must run with the defaults. + +## Discussion + +### Hash Compatibility + +This section discusses possible solutions to the problem of maintaining backwards-compatibility +of hashed parameters while adding new parameters. + +#### Never Hash Defaults + +One solution to the problem of backwards-compatibility is to never include parameters +in the hash if the are using the default value. This means that blocks produced +before the parameters existed will have implicitly been created with the defaults. +This works because any software with newer versions of Tendermint must be using the +defaults for new parameters when validating old blocks since the defaults can not +have been updated until a height at which the parameters existed. + +#### Only Update HashedParams on Hash-Breaking Releases + +An alternate solution to never hashing defaults is to not update the hashed +parameters on non-hash-breaking releases. This means that when new consensus +parameters are added to Tendermint, there may be a release that makes use of the +parameters but does not verify that they are the same across all validators by +referencing them in the hash. This seems reasonably safe given the fact that +only a very far subset of the consensus parameters are currently verified at all. + +#### Version The Consensus Parameter Hash Scheme + +The upcoming work on [soft upgrades](https://github.com/tendermint/spec/pull/222) +proposes applying different hashing rules depending on the active block version. +The consensus parameter hash could be versioned in the same way. When different +block versions are used, a different set of consensus parameters will be included +in the hash. + +### Developer Defined Values + +This section discusses possible solutions to the problem of allowing application +developers to define values for the new parameters during the upgrade that adds +the parameters. + +#### Using `InitChain` for New Values + +One solution to the problem of allowing application developers to define values +for new consensus parameters is to call the `InitChain` ABCI method on application +startup and fetch the value for any new consensus parameters. The [response object][init-chain-response] +contains a field for `ConsensusParameter` updates so this may serve as a natural place +to put this logic. + +This poses a few difficulties. Nodes replaying old blocks while running new +software do not ever call `InitChain` after the initial time. They will therefore +not have a way to determine that the parameters changed at some height by using a +call to `InitChain`. The `EndBlock` response is how parameter changes at a height +are currently communicated to Tendermint and conflating these cases seems risky. + +#### Force Defaults For Single Height + +An alternate option is to not use `InitChain` and instead require chains to use the +default values of the new parameters for a single height. + +As documented in the upcoming [ADR-74][adr-74], popular chains often simply use the default +values. Additionally, great care is being taken to ensure that logic governed by upcoming +consensus parameters is not liveness-breaking. This means that, at worst-case, +chains will experience a single slow height while waiting for the new values to +by applied. + +#### Add a new `UpgradeChain` method + +An additional method for allowing chains to update the consensus parameters that +do not yet exist is to add a new `UpgradeChain` method to `ABCI`. The upgrade chain +method would be called when the chain detects that the version of block that it +is about to produce does not match the previous block. This method would be called +after `EndBlock` and would return the set of consensus parameters to use at the +next height. It would therefore give an application the chance to set the new +consensus parameters before running a height with these new parameter. + +### References + +[hashed-params]: https://github.com/tendermint/tendermint/blob/0ae974e63911804d4a2007bd8a9b3ad81d6d2a90/types/params.go#L49 +[init-chain-response]: https://github.com/tendermint/tendermint/blob/0ae974e63911804d4a2007bd8a9b3ad81d6d2a90/abci/types/types.pb.go#L1616 +[adr-74]: https://github.com/tendermint/tendermint/pull/7503 diff --git a/cometbft/v0.38/docs/rfc/tendermint-core/rfc-010-p2p-light-client.rst b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-010-p2p-light-client.rst new file mode 100644 index 00000000..b5f46558 --- /dev/null +++ b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-010-p2p-light-client.rst @@ -0,0 +1,145 @@ +================================== +RFC 010: Peer to Peer Light Client +================================== + +Changelog +--------- + +- 2022-01-21: Initial draft (@tychoish) + +Abstract +-------- + +The dependency on access to the RPC system makes running or using the light +client more complicated than it should be, because in practice node operators +choose to restrict access to these end points (often correctly.) There is no +deep dependency for the light client on the RPC system, and there is a +persistent notion that "make a p2p light client" is a solution to this +operational limitation. This document explores the implications and +requirements of implementing a p2p-based light client, as well as the +possibilities afforded by this implementation. + +Background +---------- + +High Level Design +~~~~~~~~~~~~~~~~~ + +From a high level, the light client P2P implementation, is relatively straight +forward, but is orthogonal to the P2P-backed statesync implementation that +took place during the 0.35 cycle. The light client only really needs to be +able to request (and receive) a `LightBlock` at a given height. To support +this, a new Reactor would run on every full node and validator which would be +able to service these requests. The workload would be entirely +request-response, and the implementation of the reactor would likely be very +straight forward, and the implementation of the provider is similarly +relatively simple. + +The complexity of the project focuses around peer discovery, handling when +peers disconnect from the light clients, and how to change the current P2P +code to appropriately handle specialized nodes. + +I believe it's safe to assume that much of the current functionality of the +current ``light`` mode would *not* need to be maintained: there is no need to +proxy the RPC endpoints over the P2P layer and there may be no need to run a +node/process for the p2p light client (e.g. all use of this will be as a +client.) + +The ability to run light clients using the RPC system will continue to be +maintained. + +LibP2P +~~~~~~ + +While some aspects of the P2P light client implementation are orthogonal to +LibP2P project, it's useful to think about the ways that these efforts may +combine or interact. + +We expect to be able to leverage libp2p tools to provide some kind of service +discovery for tendermint-based networks. This means that it will be possible +for the p2p stack to easily identify specialized nodes, (e.g. light clients) +thus obviating many of the design challenges with providing this feature in +the context of the current stack. + +Similarly, libp2p makes it possible for a project to be able back their non-Go +light clients, without the major task of first implementing Tendermint's p2p +connection handling. We should identify if there exist users (e.g. the go IBC +relayer, it's maintainers, and operators) who would be able to take advantage +of p2p light client, before switching to libp2p. To our knowledge there are +limited implementations of this p2p protocol (a simple implementation without +secret connection support exists in rust but it has not been used in +production), and it seems unlikely that a team would implement this directly +ahead of its impending removal. + +User Cases +~~~~~~~~~~ + +This RFC makes a few assumptions about the use cases and users of light +clients in tendermint. + +The most active and delicate use cases for light clients is in the +implementation of the IBC relayer. Thus, we expect that providing P2P light +clients might increase the reliability of relayers and reduce the cost of +running a relayer, because relayer operators won't have to decide between rely +on public RPC endpoints (unreliable) or running their own full nodes +(expensive.) This also assumes that there are *no* other uses of the RPC in +the relayer, and unless the relayers have the option of dropping all RPC use, +it's unclear if a P2P light client will actually be able to successfully +remove the dependency on the RPC system. + +Given that the primary relayer implementation is Hermes (rust,) it might be +safe to deliver a version of Tendermint that adds a light client rector in +the full nodes, but that does not provide an implementation of a Go light +client. This either means that the rust implementation would need support for +the legacy P2P connection protocol or wait for the libp2p implementation. + +Client side light client (e.g. wallets, etc.) users may always want to use (a +subset) of the RPC rather than connect to the P2P network for an ephemeral +use. + +Discussion +---------- + +Implementation Questions +~~~~~~~~~~~~~~~~~~~~~~~~ + +Most of the complication in the is how to have a long lived light client node +that *only* runs the light client reactor, as this raises a few questions: + +- would users specify a single P2P node to connect to when creating a light + client or would they also need/want to discover peers? + + - **answer**: most light client use cases won't care much about selecting + peers (and those that do can either disable PEX and specify persistent + peers, *or* use the RPC light client.) + +- how do we prevent full nodes and validators from allowing their peer slots, + which are typically limited, from filling with light clients? If + light-clients aren't limited, how do we prevent light clients from consuming + resources on consensus nodes? + + - **answer**: I think we can institute an internal cap on number of light + client connections to accept and also elide light client nodes from PEX + (pre-libp2p, if we implement this.) I believe that libp2p should provide + us with the kind of service discovery semantics for network connectivity + that would obviate this issue. + +- when a light client disconnects from its peers will it need to reset its + internal state (cache)? does this change if it connects to the same peers? + + - **answer**: no, the internal state only needs to be reset if the light + client detects an invalid block or other divergence, and changing + witnesses--which will be more common with a p2p light client--need not + invalidate the cache. + +These issues are primarily present given that the current peer management later +does not have a particularly good service discovery mechanism nor does it have +a very sophisticated way of identifying nodes of different types or modes. + +Report Evidence +~~~~~~~~~~~~~~~ + +The current light client implementation currently has the ability to report +observed evidence. Either the notional light client reactor needs to be able +to handle these kinds of requests *or* all light client nodes need to also run +the evidence reactor. This could be configured at runtime. diff --git a/cometbft/v0.38/docs/rfc/tendermint-core/rfc-011-delete-gas.md b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-011-delete-gas.md new file mode 100644 index 00000000..854ed739 --- /dev/null +++ b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-011-delete-gas.md @@ -0,0 +1,160 @@ +# RFC 011: Remove Gas From Tendermint + +## Changelog + +- 03-Feb-2022: Initial draft (@williambanfield). +- 10-Feb-2022: Update in response to feedback (@williambanfield). +- 11-Feb-2022: Add reflection on MaxGas during consensus (@williambanfield). + +## Abstract + +In the v0.25.0 release, Tendermint added a mechanism for tracking 'Gas' in the mempool. +At a high level, Gas allows applications to specify how much it will cost the network, +often in compute resources, to execute a given transaction. While such a mechanism is common +in blockchain applications, it is not generalizable enough to be a maintained as a part +of Tendermint. This RFC explores the possibility of removing the concept of Gas from +Tendermint while still allowing applications the power to control the contents of +blocks to achieve similar goals. + +## Background + +The notion of Gas was included in the original Ethereum whitepaper and exists as +an important feature of the Ethereum blockchain. + +The [whitepaper describes Gas][eth-whitepaper-messages] as an Anti-DoS mechanism. The Ethereum Virtual Machine +provides a Turing complete execution platform. Without any limitations, malicious +actors could waste computation resources by directing the EVM to perform large +or even infinite computations. Gas serves as a metering mechanism to prevent this. + +Gas appears to have been added to Tendermint multiple times, initially as part of +a now defunct `/vm` package, and in its most recent iteration [as part of v0.25.0][gas-add-pr] +as a mechanism to limit the transactions that will be included in the block by an additional +parameter. + +Gas has gained adoption within the Cosmos ecosystem [as part of the Cosmos SDK][cosmos-sdk-gas]. +The SDK provides facilities for tracking how much 'Gas' a transaction is expected to take +and a mechanism for tracking how much gas a transaction has already taken. + +Non-SDK applications also make use of the concept of Gas. Anoma appears to implement +[a gas system][anoma-gas] to meter the transactions it executes. + +While the notion of gas is present in projects that make use of Tendermint, it is +not a concern of Tendermint's. Tendermint's value and goal is producing blocks +via a distributed consensus algorithm. Tendermint relies on the application specific +code to decide how to handle the transactions Tendermint has produced (or if the +application wants to consider them at all). Gas is an application concern. + +Our implementation of Gas is not currently enforced by consensus. Our current validation check that +occurs during block propagation does not verify that the block is under the configured `MaxGas`. +Ensuring that the transactions in a proposed block do not exceed `MaxGas` would require +input from the application during propagation. The `ProcessProposal` method introduced +as part of ABCI++ would enable such input but would further entwine Tendermint and +the application. The issue of checking `MaxGas` during block propagation is important +because it demonstrates that the feature as it currently exists is not implemented +as fully as it perhaps should be. + +Our implementation of Gas is causing issues for node operators and relayers. At +the moment, transactions that overflow the configured 'MaxGas' can be silently rejected +from the mempool. Overflowing MaxGas is the _only_ way that a transaction can be considered +invalid that is not directly a result of failing the `CheckTx`. Operators, and the application, +do not know that a transaction was removed from the mempool for this reason. A stateless check +of this nature is exactly what `CheckTx` exists for and there is no reason for the mempool +to keep track of this data separately. A special [MempoolError][add-mempool-error] field +was added in v0.35 to communicate to clients that a transaction failed after `CheckTx`. +While this should alleviate the pain for operators wishing to understand if their +transaction was included in the mempool, it highlights that the abstraction of +what is included in the mempool is not currently well defined. + +Removing Gas from Tendermint and the mempool would allow for the mempool to be a better +abstraction: any transaction that arrived at `CheckTx` and passed the check will either be +a candidate for a later block or evicted after a TTL is reached or to make room for +other, higher priority transactions. All other transactions are completely invalid and can be discarded forever. + +Removing gas will not be completely straightforward. It will mean ensuring that +equivalent functionality can be implemented outside of the mempool using the mempool's API. + +## Discussion + +This section catalogs the functionality that will need to exist within the Tendermint +mempool to allow Gas to be removed and replaced by application-side bookkeeping. + +### Requirement: Provide Mempool Tx Sorting Mechanism + +Gas produces a market for inclusion in a block. On many networks, a [gas fee][cosmos-sdk-fees] is +included in pending transactions. This fee indicates how much a user is willing to +pay per unit of execution and the fees are distributed to validators. + +Validators wishing to extract higher gas fees are incentivized to include transactions +with the highest listed gas fees into each block. This produces a natural ordering +of the pending transactions. Applications wishing to implement a gas mechanism need +to be able to order the transactions in the mempool. This can trivially be accomplished +by sorting transactions using the `priority` field available to applications as part of +v0.35's `ResponseCheckTx` message. + +### Requirement: Allow Application-Defined Block Resizing + +When creating a block proposal, Tendermint pulls a set of possible transactions out of +the mempool to include in the next block. Tendermint uses MaxGas to limit the set of transactions +it pulls out of the mempool fetching a set of transactions whose sum is less than MaxGas. + +By removing gas tracking from Tendermint's mempool, Tendermint will need to provide a way for +applications to determine an acceptable set of transactions to include in the block. + +This is what the new ABCI++ `PrepareProposal` method is useful for. Applications +that wish to limit the contents of a block by an application-defined limit may +do so by removing transactions from the proposal it is passed during `PrepareProposal`. +Applications wishing to reach parity with the current Gas implementation may do +so by creating an application-side limit: filtering out transactions from +`PrepareProposal` the cause the proposal the exceed the maximum gas. Additionally, +applications can currently opt to have all transactions in the mempool delivered +during `PrepareProposal` by passing `-1` for `MaxGas` and `MaxBytes` into +[ReapMaxBytesMaxGas][reap-max-bytes-max-gas]. + +### Requirement: Handle Transaction Metadata + +Moving the gas mechanism into applications adds an additional piece of complexity +to applications. The application must now track how much gas it expects a transaction +to consume. The mempool currently handles this bookkeeping responsibility and uses the estimated +gas to determine the set of transactions to include in the block. In order to task +the application with keeping track of this metadata, we should make it easier for the +application to do so. In general, we'll want to keep only one copy of this type +of metadata in the program at a time, either in the application or in Tendermint. + +The following sections are possible solutions to the problem of storing transaction +metadata without duplication. + +#### Metadata Handling: EvictTx Callback + +A possible approach to handling transaction metadata is by adding a new `EvictTx` +ABCI method. Whenever the mempool is removing a transaction, either because it has +reached its TTL or because it failed `RecheckTx`, `EvictTx` would be called with +the transaction hash. This would indicate to the application that it could free any +metadata it was storing about the transaction such as the computed gas fee. + +Eviction callbacks are pretty common in caching systems, so this would be very +well-worn territory. + +#### Metadata Handling: Application-Specific Metadata Field(s) + +An alternative approach to handling transaction metadata would be would be the +addition of a new application-metadata field in the `ResponseCheckTx`. This field +would be a protocol buffer message whose contents were entirely opaque to Tendermint. +The application would be responsible for marshalling and unmarshalling whatever data +it stored in this field. During `PrepareProposal`, the application would be passed +this metadata along with the transaction, allowing the application to use it to perform +any necessary filtering. + +If either of these proposed metadata handling techniques are selected, it's likely +useful to enable applications to gossip metadata along with the transaction it is +gossiping. This could easily take the form of an opaque proto message that is +gossiped along with the transaction. + +## References + +[eth-whitepaper-messages]: https://ethereum.org/en/whitepaper/#messages-and-transactions +[gas-add-pr]: https://github.com/tendermint/tendermint/pull/2360 +[cosmos-sdk-gas]: https://github.com/cosmos/cosmos-sdk/blob/c00cedb1427240a730d6eb2be6f7cb01f43869d3/docs/basics/gas-fees.md +[cosmos-sdk-fees]: https://github.com/cosmos/cosmos-sdk/blob/c00cedb1427240a730d6eb2be6f7cb01f43869d3/docs/basics/tx-lifecycle.md#gas-and-fees +[anoma-gas]: https://github.com/anoma/anoma/blob/6974fe1532a59db3574fc02e7f7e65d1216c1eb2/docs/src/specs/ledger.md#transaction-execution +[reap-max-bytes-max-gas]: https://github.com/tendermint/tendermint/blob/1ac58469f32a98f1c0e2905ca1773d9eac7b7103/internal/mempool/types.go#L45 +[add-mempool-error]: https://github.com/tendermint/tendermint/blob/205bfca66f6da1b2dded381efb9ad3792f9404cf/rpc/coretypes/responses.go#L239 diff --git a/cometbft/v0.38/docs/rfc/tendermint-core/rfc-012-custom-indexing.md b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-012-custom-indexing.md new file mode 100644 index 00000000..489bcccc --- /dev/null +++ b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-012-custom-indexing.md @@ -0,0 +1,351 @@ +# RFC 012: Event Indexing Revisited + +## Changelog + +- 11-Feb-2022: Add terminological notes. +- 10-Feb-2022: Updated from review feedback. +- 07-Feb-2022: Initial draft (@creachadair) + +## Abstract + +A Tendermint node allows ABCI events associated with block and transaction +processing to be "indexed" into persistent storage. The original Tendermint +implementation provided a fixed, built-in [proprietary indexer][kv-index] for +such events. + +In response to user requests to customize indexing, [ADR 065][adr065] +introduced an "event sink" interface that allows developers (at least in +theory) to plug in alternative index storage. + +Although ADR-065 was a good first step toward customization, its implementation +model does not satisfy all the user requirements. Moreover, this approach +leaves some existing technical issues with indexing unsolved. + +This RFC documents these concerns, and discusses some potential approaches to +solving them. This RFC does _not_ propose a specific technical decision. It is +meant to unify and focus some of the disparate discussions of the topic. + + +## Background + +We begin with some important terminological context. The term "event" in +Tendermint can be confusing, as the same word is used for multiple related but +distinct concepts: + +1. **ABCI Events** refer to the key-value metadata attached to blocks and + transactions by the application. These values are represented by the ABCI + `Event` protobuf message type. + +2. **Consensus Events** refer to the data published by the Tendermint node to + its pubsub bus in response to various consensus state transitions and other + important activities, such as round updates, votes, transaction delivery, + and block completion. + +This confusion is compounded because some "consensus event" values also have +"ABCI event" metadata attached to them. Notably, block and transaction items +typically have ABCI metadata assigned by the application. + +Indexers and RPC clients subscribed to the pubsub bus receive **consensus +events**, but they identify which ones to care about using query expressions +that match against the **ABCI events** associated with them. + +In the discussion that follows, we will use the term **event item** to refer to +a datum published to or received from the pubsub bus, and **ABCI event** or +**event metadata** to refer to the key/value annotations. + +**Indexing** in this context means recording the association between certain +ABCI metadata and the blocks or transactions they're attached to. The ABCI +metadata typically carry application-specific details like sender and recipient +addresses, catgory tags, and so forth, that are not part of consensus but are +used by UI tools to find and display transactions of interest. + +The consensus node records the blocks and transactions as part of its block +store, but does not persist the application metadata. Metadata persistence is +the task of the indexer, which can be (optionally) enabled by the node +operator. + +### History + +The [original indexer][kv-index] built in to Tendermint stored index data in an +embedded [`tm-db` database][tmdb] with a proprietary key layout. +In [ADR 065][adr065], we noted that this implementation has both performance +and scaling problems under load. Moreover, the only practical way to query the +index data is via the [query filter language][query] used for event +subscription. [Issue #1161][i1161] appears to be a motivational context for that ADR. + +To mitigate both of these concerns, we introduced the [`EventSink`][esink] +interface, combining the original transaction and block indexer interfaces +along with some service plumbing. Using this interface, a developer can plug +in an indexer that uses a more efficient storage engine, and provides a more +expressive query language. As a proof-of-concept, we built a [PostgreSQL event +sink][psql] that exports data to a [PostgreSQL database][postgres]. + +Although this approach addressed some of the immediate concerns, there are +several issues for custom indexing that have not been fully addressed. Here we +will discuss them in more detail. + +For further context, including links to user reports and related work, see also +the [Pluggable custom event indexing tracking issue][i7135] issue. + +### Issue 1: Tight Coupling + +The `EventSink` interface supports multiple implementations, but plugging in +implementations still requires tight integration with the node. In particular: + +- Any custom indexer must either be written in Go and compiled in to the + Tendermint binary, or the developer must write a Go shim to communicate with + the implementation and build that into the Tendermint binary. + +- This means to support a custom indexer, it either has to be integrated into + the Tendermint core repository, or every installation that uses that indexer + must fetch or build a patched version of Tendermint. + +The problem with integrating indexers into Tendermint Core is that every user +of Tendermint Core takes a dependency on all supported indexers, including +those they never use. Even if the unused code is disabled with build tags, +users have to remember to do this or potentially be exposed to security issues +that may arise in any of the custom indexers. This is a risk for Tendermint, +which is a trust-critical component of all applications built on it. + +The problem with _not_ integrating indexers into Tendermint Core is that any +developer who wants to use a particular indexer must now fetch or build a +patched version of the core code that includes the custom indexer. Besides +being inconvenient, this makes it harder for users to upgrade their node, since +they need to either re-apply their patches directly or wait for an intermediary +to do it for them. + +Even for developers who have written their applications in Go and link with the +consensus node directly (e.g., using the [Cosmos SDK][sdk]), these issues add a +potentially significant complication to the build process. + +### Issue 2: Legacy Compatibility + +The `EventSink` interface retains several limitations of the original +proprietary indexer. These include: + +- The indexer has no control over which event items are reported. Only the + exact block and transaction events that were reported to the original indexer + are reported to a custom indexer. + +- The interface requires the implementation to define methods for the legacy + search and query API. This requirement comes from the integation with the + [event subscription RPC API][event-rpc], but actually supporting these + methods is not trivial. + +At present, only the original KV indexer implements the query methods. Even the +proof-of-concept PostgreSQL implementation simply reports errors for all calls +to these methods. + +Even for a plugin written in Go, implementing these methods "correctly" would +require parsing and translating the custom query language over whatever storage +platform the indexer uses. + +For a plugin _not_ written in Go, even beyond the cost of integration the +developer would have to re-implement the entire query language. + +### Issue 3: Indexing Delays Consensus + +Within the node, indexing hooks in to the same internal pubsub dispatcher that +is used to export event items to the [event subscription RPC API][event-rpc]. +In contrast with RPC subscribers, however, indexing is a "privileged" +subscriber: If an RPC subscriber is "too slow", the node may terminate the +subscription and disconnect the client. That means that RPC subscribers may +lose (miss) event items. The indexer, however, is "unbuffered", and the +publisher will never drop or disconnect from it. If the indexer is slow, the +publisher will block until it returns, to ensure that no event items are lost. + +In practice, this means that the performance of the indexer has a direct effect +on the performance of the consensus node: If the indexer is slow or stalls, it +will slow or halt the progress of consensus. Users have already reported this +problem even with the built-in indexer (see, for example, [#7247][i7247]). +Extending this concern to arbitrary user-defined custom indexers gives that +risk a much larger surface area. + + +## Discussion + +It is not possible to simultaneously guarantee that publishing event items will +not delay consensus, and also that all event items of interest are always +completely indexed. + +Therefore, our choice is between eliminating delay (and minimizing loss) or +eliminating loss (and minimizing delay). Currently, we take the second +approach, which has led to user complaints about consensus delays due to +indexing and subscription overhead. + +- If we agree that consensus performance supersedes index completeness, our + design choices are to constrain the likelihood and frequency of missing event + items. + +- If we decide that consensus performance is more important than index + completeness, our option is to minimize overhead on the event delivery path + and document that indexer plugins constrain the rate of consensus. + +Since we have user reports requesting both properties, we have to choose one or +the other. Since the primary job of the consensus engine is to correctly, +robustly, reliablly, and efficiently replicate application state across the +network, I believe the correct choice is to favor consensus performance. + +An important consideration for this decision is that a node does not index +application metadata separately: If indexing is disabled, there is no built-in +mechanism to go back and replay or reconstruct the data that an indexer would +have stored. The node _does_ store the blockchain itself (i.e., the blocks and +their transactions), so potentially some use cases currently handled by the +indexer could be handled by the node. For example, allowing clients to ask +whether a given transaction ID has been committed to a block could in principle +be done without an indexer, since it does not depend on application metadata. + +Inevitably, a question will arise whether we could implement both strategies +and toggle between them with a flag. That would be a worst-case scenario, +requiring us to maintain the complexity of two very-different operational +concerns. If our goal is that Tendermint should be as simple, efficient, and +trustworthy as posible, there is not a strong case for making these options +configurable: We should pick a side and commit to it. + +### Design Principles + +Although there is no unique "best" solution to the issues described above, +there are some specific principles that a solution should include: + +1. **A custom indexer should not require integration into Tendermint core.** A + developer or node operator can create, build, deploy, and use a custom + indexer with a stock build of the Tendermint consensus node. + +2. **Custom indexers cannot stall consensus.** An indexer that is slow or + stalls cannot slow down or prevent core consensus from making progress. + + The plugin interface must give node operators control over the tolerances + for acceptable indexer performance, and the means to detect when indexers + are falling outside those tolerances, but indexer failures should "fail + safe" with respect to consensus (even if that means the indexer may miss + some data, in sufficiently-extreme circumstances). + +3. **Custom indexers control which event items they index.** A custom indexer + is not limited to only the current transaction and block events, but can + observe any event item published by the node. + +4. **Custom indexing is forward-compatible.** Adding new event item types or + metadata to the consensus node should not require existing custom indexers + to be rebuilt or modified, unless they want to take advantage of the new + data. + +5. **Indexers are responsible for answering queries.** An indexer plugin is not + required to support the legacy query filter language, nor to be compatible + with the legacy RPC endpoints for accessing them. Any APIs for clients to + query a custom index are the responsibility of the indexer, not the node. + +### Open Questions + +Given the constraints outlined above, there are important design questions we +must answer to guide any specific changes: + +1. **What is an acceptable probability that, given sufficiently extreme + operational issues, an indexer might miss some number of events?** + + There are two parts to this question: One is what constitutes an extreme + operational problem, the other is how likely we are to miss some number of + events items. + + - If the consensus is that no event item must ever be missed, no matter how + bad the operational circumstances, then we _must_ accept that indexing can + slow or halt consensus arbitrarily. It is impossible to guarantee complete + index coverage without potentially unbounded delays. + + - Otherwise, how much data can we afford to lose and how often? For example, + if we can ensure no event item will be lost unless the indexer halts for + at least five minutes, is that acceptable? What probabilities and time + ranges are reasonable for real production environments? + +2. **What level of operational overhead is acceptable to impose on node + operators to support indexing?** + + Are node operators willing to configure and run custom indexers as sidecar + type processes alongside a node? How much indexer setup above and beyond the + work of setting up the underlying node in isolation is tractable in + production networks? + + The answer to this question also informs the question of whether we should + keep an "in-process" indexing option, and to what extent that option needs + to satisfy the suggested design principles. + + Relatedly, to what extent do we need to be concerned about the cost of + encoding and sending event items to an external process (e.g., as JSON blobs + or protobuf wire messages)? Given that the node already encodes event items + as JSON for subscription purposes, the overhead would be negligible for the + node itself, but the indexer would have to decode to process the results. + +3. **What (if any) query APIs does the consensus node need to export, + independent of the indexer implementation?** + + One typical example is whether the node should be able to answer queries + like "is this transaction ID in a block?" Currently, a node cannot answer + this query _unless_ it runs the built-in KV indexer. Does the node need to + continue to support that query even for nodes that disable the KV indexer, + or which use a custom indexer? + +### Informal Design Intent + +The design principles described above implicate several components of the +Tendermint node, beyond just the indexer. In the context of [ADR 075][adr075], +we are re-working the RPC event subscription API to improve some of the UX +issues discussed above for RPC clients. It is our expectation that a solution +for pluggable custom indexing will take advantage of some of the same work. + +On that basis, the design approach I am considering for custom indexing looks +something like this (subject to refinement): + +1. A custom indexer runs as a separate process from the node. + +2. The indexer subscribes to event items via the ADR 075 events API. + + This means indexers would receive event payloads as JSON rather than + protobuf, but since we already have to support JSON encoding for the RPC + interface anyway, that should not increase complexity for the node. + +3. The existing PostgreSQL indexer gets reworked to have this form, and no + longer built as part of the Tendermint core binary. + + We can retain the code in the core repository as a proof-of-concept, or + perhaps create a separate repository with contributed indexers and move it + there. + +4. (Possibly) Deprecate and remove the legacy KV indexer, or disable it by + default. If we decide to remove it, we can also remove the legacy RPC + endpoints for querying the KV indexer. + + If we plan to do this, we should also investigate providing a way for + clients to query whether a given transaction ID has landed in a block. That + serves a common need, and currently _only_ works if the KV indexer is + enabled, but could be addressed more simply using the other data a node + already has stored, without having to answer more general queries. + + +## References + +- [ADR 065: Custom Event Indexing][adr065] +- [ADR 075: RPC Event Subscription Interface][adr075] +- [Cosmos SDK][sdk] +- [Event subscription RPC][event-rpc] +- [KV transaction indexer][kv-index] +- [Pluggable custom event indexing][i7135] (#7135) +- [PostgreSQL event sink][psql] + - [PostgreSQL database][postgres] +- [Query filter language][query] +- [Stream events to postgres for indexing][i1161] (#1161) +- [Unbuffered event subscription slow down the consensus][i7247] (#7247) +- [`EventSink` interface][esink] +- [`tm-db` library][tmdb] + +[adr065]: https://github.com/tendermint/tendermint/blob/main/docs/architecture/adr-065-custom-event-indexing.md +[adr075]: https://github.com/tendermint/tendermint/blob/main/docs/architecture/adr-075-rpc-subscription.md +[esink]: https://pkg.go.dev/github.com/tendermint/tendermint/internal/state/indexer#EventSink +[event-rpc]: https://docs.tendermint.com/v0.34/rpc/#/Websocket/subscribe +[i1161]: https://github.com/tendermint/tendermint/issues/1161 +[i7135]: https://github.com/tendermint/tendermint/issues/7135 +[i7247]: https://github.com/tendermint/tendermint/issues/7247 +[kv-index]: https://github.com/tendermint/tendermint/blob/main/state/indexer/block/kv +[postgres]: https://postgresql.org/ +[psql]: https://github.com/tendermint/tendermint/tree/main/state/indexer/sink/psql +[query]: https://pkg.go.dev/github.com/tendermint/tendermint/internal/pubsub/query/syntax +[sdk]: https://github.com/cosmos/cosmos-sdk +[tmdb]: https://pkg.go.dev/github.com/tendermint/tm-db#DB diff --git a/cometbft/v0.38/docs/rfc/tendermint-core/rfc-013-abci++.md b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-013-abci++.md new file mode 100644 index 00000000..6e83c9aa --- /dev/null +++ b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-013-abci++.md @@ -0,0 +1,253 @@ +# RFC 013: ABCI++ + +## Changelog + +- 2020-01-11: initialized +- 2022-02-11: Migrate RFC to tendermint repo (Originally [RFC 004](https://github.com/tendermint/spec/pull/254)) + +## Author(s) + +- Dev (@valardragon) +- Sunny (@sunnya97) + +## Context + +ABCI is the interface between the consensus engine and the application. +It defines when the application can talk to consensus during the execution of a blockchain. +At the moment, the application can only act at one phase in consensus, immediately after a block has been finalized. + +This restriction on the application prohibits numerous features for the application, including many scalability improvements that are now better understood than when ABCI was first written. +For example, many of the scalability proposals can be boiled down to "Make the miner / block proposers / validators do work, so the network does not have to". +This includes optimizations such as tx-level signature aggregation, state transition proofs, etc. +Furthermore, many new security properties cannot be achieved in the current paradigm, as the application cannot enforce validators do more than just finalize txs. +This includes features such as threshold cryptography, and guaranteed IBC connection attempts. +We propose introducing three new phases to ABCI to enable these new features, and renaming the existing methods for block execution. + +#### Prepare Proposal phase + +This phase aims to allow the block proposer to perform more computation, to reduce load on all other full nodes, and light clients in the network. +It is intended to enable features such as batch optimizations on the transaction data (e.g. signature aggregation, zk rollup style validity proofs, etc.), enabling stateless blockchains with validator provided authentication paths, etc. + +This new phase will only be executed by the block proposer. The application will take in the block header and raw transaction data output by the consensus engine's mempool. It will then return block data that is prepared for gossip on the network, and additional fields to include into the block header. + +#### Process Proposal Phase + +This phase aims to allow applications to determine validity of a new block proposal, and execute computation on the block data, prior to the blocks finalization. +It is intended to enable applications to reject block proposals with invalid data, and to enable alternate pipelined execution models. (Such as Ethereum-style immediate execution) + +This phase will be executed by all full nodes upon receiving a block, though on the application side it can do more work in the even that the current node is a validator. + +#### Vote Extension Phase + +This phase aims to allow applications to require their validators do more than just validate blocks. +Example usecases of this include validator determined price oracles, validator guaranteed IBC connection attempts, and validator based threshold crypto. + +This adds an app-determined data field that every validator must include with their vote, and these will thus appear in the header. + +#### Rename {BeginBlock, [DeliverTx], EndBlock} to FinalizeBlock + +The prior phases gives the application more flexibility in their execution model for a block, and they obsolete the current methods for how the consensus engine relates the block data to the state machine. Thus we refactor the existing methods to better reflect what is happening in the new ABCI model. + +This rename doesn't on its own enable anything new, but instead improves naming to clarify the expectations from the application in this new communication model. The existing ABCI methods `BeginBlock, [DeliverTx], EndBlock` are renamed to a single method called `FinalizeBlock`. + +#### Summary + +We include a more detailed list of features / scaling improvements that are blocked, and which new phases resolve them at the end of this document. + + +On the top is the existing definition of ABCI, and on the bottom is the proposed ABCI++. + +## Proposal + +Below we suggest an API to add these three new phases. +In this document, sometimes the final round of voting is referred to as precommit for clarity in how it acts in the Tendermint case. + +### Prepare Proposal + +*Note, APIs in this section will change after Vote Extensions, we list the adjusted APIs further in the proposal.* + +The Prepare Proposal phase allows the block proposer to perform application-dependent work in a block, to lower the amount of work the rest of the network must do. This enables batch optimizations to a block, which has been empirically demonstrated to be a key component for scaling. This phase introduces the following ABCI method + +```rust +fn PrepareProposal(Block) -> BlockData +``` + +where `BlockData` is a type alias for however data is internally stored within the consensus engine. In Tendermint Core today, this is `[]Tx`. + +The application may read the entire block proposal, and mutate the block data fields. Mutated transactions will still get removed from the mempool later on, as the mempool rechecks all transactions after a block is executed. + +The `PrepareProposal` API will be modified in the vote extensions section, for allowing the application to modify the header. + +### Process Proposal + +The Process Proposal phase sends the block data to the state machine, prior to running the last round of votes on the state machine. This enables features such as allowing validators to reject a block according to whether state machine deems it valid, and changing block execution pipeline. + +We introduce three new methods, + +```rust +fn VerifyHeader(header: Header, isValidator: bool) -> ResponseVerifyHeader {...} +fn ProcessProposal(block: Block) -> ResponseProcessProposal {...} +fn RevertProposal(height: usize, round: usize) {...} +``` + +where + +```rust +struct ResponseVerifyHeader { + accept_header: bool, + evidence: Vec +} +struct ResponseProcessProposal { + accept_block: bool, + evidence: Vec +} +``` + +Upon receiving a block header, every validator runs `VerifyHeader(header, isValidator)`. The reason for why `VerifyHeader` is split from `ProcessProposal` is due to the later sections for Preprocess Proposal and Vote Extensions, where there may be application dependent data in the header that must be verified before accepting the header. +If the returned `ResponseVerifyHeader.accept_header` is false, then the validator must precommit nil on this block, and reject all other precommits on this block. `ResponseVerifyHeader.evidence` is appended to the validators local `EvidencePool`. + +Upon receiving an entire block proposal (in the current implementation, all "block parts"), every validator runs `ProcessProposal(block)`. If the returned `ResponseProcessProposal.accept_block` is false, then the validator must precommit nil on this block, and reject all other precommits on this block. `ResponseProcessProposal.evidence` is appended to the validators local `EvidencePool`. + +Once a validator knows that consensus has failed to be achieved for a given block, it must run `RevertProposal(block.height, block.round)`, in order to signal to the application to revert any potentially mutative state changes it may have made. In Tendermint, this occurs when incrementing rounds. + +**RFC**: How do we handle the scenario where honest node A finalized on round x, and honest node B finalized on round x + 1? (e.g. when 2f precommits are publicly known, and a validator precommits themself but doesn't broadcast, but they increment rounds) Is this a real concern? The state root derived could change if everyone finalizes on round x+1, not round x, as the state machine can depend non-uniformly on timestamp. + +The application is expected to cache the block data for later execution. + +The `isValidator` flag is set according to whether the current node is a validator or a full node. This is intended to allow for beginning validator-dependent computation that will be included later in vote extensions. (An example of this is threshold decryptions of ciphertexts.) + +### DeliverTx rename to FinalizeBlock + +After implementing `ProcessProposal`, txs no longer need to be delivered during the block execution phase. Instead, they are already in the state machine. Thus `BeginBlock, DeliverTx, EndBlock` can all be replaced with a single ABCI method for `ExecuteBlock`. Internally the application may still structure its method for executing the block as `BeginBlock, DeliverTx, EndBlock`. However, it is overly restrictive to enforce that the block be executed after it is finalized. There are multiple other, very reasonable pipelined execution models one can go for. So instead we suggest calling this succession of methods `FinalizeBlock`. We propose the following API + +Replace the `BeginBlock, DeliverTx, EndBlock` ABCI methods with the following method + +```rust +fn FinalizeBlock() -> ResponseFinalizeBlock +``` + +where `ResponseFinalizeBlock` has the following API, in terms of what already exists + +```rust +struct ResponseFinalizeBlock { + updates: ResponseEndBlock, + tx_results: Vec +} +``` + +`ResponseEndBlock` should then be renamed to `ConsensusUpdates` and `ResponseDeliverTx` should be renamed to `ResponseTx`. + +### Vote Extensions + +The Vote Extensions phase allow applications to force their validators to do more than just validate within consensus. This is done by allowing the application to add more data to their votes, in the final round of voting. (Namely the precommit) +This additional application data will then appear in the block header. + +First we discuss the API changes to the vote struct directly + +```rust +fn ExtendVote(height: u64, round: u64) -> (UnsignedAppVoteData, SelfAuthenticatingAppData) +fn VerifyVoteExtension(signed_app_vote_data: Vec, self_authenticating_app_vote_data: Vec) -> bool +``` + +There are two types of data that the application can enforce validators to include with their vote. +There is data that the app needs the validator to sign over in their vote, and there can be self-authenticating vote data. Self-authenticating here means that the application upon seeing these bytes, knows its valid, came from the validator and is non-malleable. We give an example of each type of vote data here, to make their roles clearer. + +- Unsigned app vote data: A use case of this is if you wanted validator backed oracles, where each validator independently signs some oracle data in their vote, and the median of these values is used on chain. Thus we leverage consensus' signing process for convenience, and use that same key to sign the oracle data. +- Self-authenticating vote data: A use case of this is in threshold random beacons. Every validator produces a threshold beacon share. This threshold beacon share can be verified by any node in the network, given the share and the validators public key (which is not the same as its consensus public key). However, this decryption share will not make it into the subsequent block's header. They will be aggregated by the subsequent block proposer to get a single random beacon value that will appear in the subsequent block's header. Everyone can then verify that this aggregated value came from the requisite threshold of the validator set, without increasing the bandwidth for full nodes or light clients. To achieve this goal, the self-authenticating vote data cannot be signed over by the consensus key along with the rest of the vote, as that would require all full nodes & light clients to know this data in order to verify the vote. + +The `CanonicalVote` struct will acommodate the `UnsignedAppVoteData` field by adding another string to its encoding, after the `chain-id`. This should not interfere with existing hardware signing integrations, as it does not affect the constant offset for the `height` and `round`, and the vote size does not have an explicit upper bound. (So adding this unsigned app vote data field is equivalent from the HSM's perspective as having a superlong chain-ID) + +**RFC**: Please comment if you think it will be fine to have elongate the message the HSM signs, or if we need to explore pre-hashing the app vote data. + +The flow of these methods is that when a validator has to precommit, Tendermint will first produce a precommit canonical vote without the application vote data. It will then pass it to the application, which will return unsigned application vote data, and self authenticating application vote data. It will bundle the `unsigned_application_vote_data` into the canonical vote, and pass it to the HSM to sign. Finally it will package the self-authenticating app vote data, and the `signed_vote_data` together, into one final Vote struct to be passed around the network. + +#### Changes to Prepare Proposal Phase + +There are many use cases where the additional data from vote extensions can be batch optimized. +This is mainly of interest when the votes include self-authenticating app vote data that be batched together, or the unsigned app vote data is the same across all votes. +To allow for this, we change the PrepareProposal API to the following + +```rust +fn PrepareProposal(Block, UnbatchedHeader) -> (BlockData, Header) +``` + +where `UnbatchedHeader` essentially contains a "RawCommit", the `Header` contains a batch-optimized `commit` and an additional "Application Data" field in its root. This will involve a number of changes to core data structures, which will be gone over in the ADR. +The `Unbatched` header and `rawcommit` will never be broadcasted, they will be completely internal to consensus. + +#### Inter-process communication (IPC) effects + +For brevity in exposition above, we did not discuss the trade-offs that may occur in interprocess communication delays that these changs will introduce. +These new ABCI methods add more locations where the application must communicate with the consensus engine. +In most configurations, we expect that the consensus engine and the application will be either statically or dynamically linked, so all communication is a matter of at most adjusting the memory model the data is layed out within. +This memory model conversion is typically considered negligible, as delay here is measured on the order of microseconds at most, whereas we face milisecond delays due to cryptography and network overheads. +Thus we ignore the overhead in the case of linked libraries. + +In the case where the consensus engine and the application are ran in separate processes, and thus communicate with a form of Inter-process communication (IPC), the delays can easily become on the order of miliseconds based upon the data sent. Thus its important to consider whats happening here. +We go through this phase by phase. + +##### Prepare proposal IPC overhead + +This requires a round of IPC communication, where both directions are quite large. Namely the proposer communicating an entire block to the application. +However, this can be mitigated by splitting up `PrepareProposal` into two distinct, async methods, one for the block IPC communication, and one for the Header IPC communication. + +Then for chains where the block data does not depend on the header data, the block data IPC communication can proceed in parallel to the prior block's voting phase. (As a node can know whether or not its the leader in the next round) + +Furthermore, this IPC communication is expected to be quite low relative to the amount of p2p gossip time it takes to send the block data around the network, so this is perhaps a premature concern until more sophisticated block gossip protocols are implemented. + +##### Process Proposal IPC overhead + +This phase changes the amount of time available for the consensus engine to deliver a block's data to the state machine. +Before, the block data for block N would be delivered to the state machine upon receiving a commit for block N and then be executed. +The state machine would respond after executing the txs and before prevoting. +The time for block delivery from the consensus engine to the state machine after this change is the time of receiving block proposal N to the to time precommit on proposal N. +It is expected that this difference is unimportant in practice, as this time is in parallel to one round of p2p communication for prevoting, which is expected to be significantly less than the time for the consensus engine to deliver a block to the state machine. + +##### Vote Extension IPC overhead + +This has a small amount of data, but does incur an IPC round trip delay. This IPC round trip delay is pretty negligible as compared the variance in vote gossip time. (the IPC delay is typically on the order of 10 microseconds) + +## Status + +Proposed + +## Consequences + +### Positive + +- Enables a large number of new features for applications +- Supports both immediate and delayed execution models +- Allows application specific data from each validator +- Allows for batch optimizations across txs, and votes + +### Negative + +- This is a breaking change to all existing ABCI clients, however the application should be able to have a thin wrapper to replicate existing ABCI behavior. + - PrepareProposal - can be a no-op + - Process Proposal - has to cache the block, but can otherwise be a no-op + - Vote Extensions - can be a no-op + - Finalize Block - Can black-box call BeginBlock, DeliverTx, EndBlock given the cached block data + +- Vote Extensions adds more complexity to core Tendermint Data Structures +- Allowing alternate alternate execution models will lead to a proliferation of new ways for applications to violate expected guarantees. + +### Neutral + +- IPC overhead considerations change, but mostly for the better + +## References + +Reference for IPC delay constants: + +### Short list of blocked features / scaling improvements with required ABCI++ Phases + +| Feature | PrepareProposal | ProcessProposal | Vote Extensions | +| :--- | :---: | :---: | :---: | +| Tx based signature aggregation | X | | | +| SNARK proof of valid state transition | X | | | +| Validator provided authentication paths in stateless blockchains | X | | | +| Immediate Execution | | X | | +| Simple soft forks | | X | | +| Validator guaranteed IBC connection attempts | | | X | +| Validator based price oracles | | | X | +| Immediate Execution with increased time for block execution | X | X | X | +| Threshold Encrypted txs | X | X | X | diff --git a/cometbft/v0.38/docs/rfc/tendermint-core/rfc-014-semantic-versioning.md b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-014-semantic-versioning.md new file mode 100644 index 00000000..0119901b --- /dev/null +++ b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-014-semantic-versioning.md @@ -0,0 +1,94 @@ +# RFC 014: Semantic Versioning + +## Changelog + +- 2021-11-19: Initial Draft +- 2021-02-11: Migrate RFC to tendermint repo (Originally [RFC 006](https://github.com/tendermint/spec/pull/365)) + +## Author(s) + +- Callum Waters @cmwaters + +## Context + +We use versioning as an instrument to hold a set of promises to users and signal when such a set changes and how. In the conventional sense of a Go library, major versions signal that the public Go API’s have changed in a breaking way and thus require the users of such libraries to change their usage accordingly. Tendermint is a bit different in that there are multiple users: application developers (both in-process and out-of-process), node operators, and external clients. More importantly, both how these users interact with Tendermint and what's important to these users differs from how users interact and what they find important in a more conventional library. + +This document attempts to encapsulate the discussions around versioning in Tendermint and draws upon them to propose a guide to how Tendermint uses versioning to make promises to its users. + +For a versioning policy to make sense, we must also address the intended frequency of breaking changes. The strictest guarantees in the world will not help users if we plan to break them with every release. + +Finally I would like to remark that this RFC only addresses the "what", as in what are the rules for versioning. The "how" of Tendermint implementing the versioning rules we choose, will be addressed in a later RFC on Soft Upgrades. + +## Discussion + +We first begin with a round up of the various users and a set of assumptions on what these users expect from Tendermint in regards to versioning: + +1. **Application Developers**, those that use the ABCI to build applications on top of Tendermint, are chiefly concerned with that API. Breaking changes will force developers to modify large portions of their codebase to accommodate for the changes. Some ABCI changes such as introducing priority for the mempool don't require any effort and can be lazily adopted whilst changes like ABCI++ may force applications to redesign their entire execution system. It's also worth considering that the API's for go developers differ to developers of other languages. The former here can use the entire Tendermint library, most notably the local RPC methods, and so the team must be wary of all public Go API's. +2. **Node Operators**, those running node infrastructure, are predominantly concerned with downtime, complexity and frequency of upgrading, and avoiding data loss. They may be also concerned about changes that may break the scripts and tooling they use to supervise their nodes. +3. **External Clients** are those that perform any of the following: + - consume the RPC endpoints of nodes like `/block` + - subscribe to the event stream + - make queries to the indexer + + This set are concerned with chain upgrades which will impact their ability to query state and block data as well as broadcast transactions. Examples include wallets and block explorers. + +4. **IBC module and relayers**. The developers of IBC and consumers of their software are concerned about changes that may affect a chain's ability to send arbitrary messages to another chain. Specifically, these users are affected by any breaking changes to the light client verification algorithm. + +Although we present them here as having different concerns, in a broader sense these user groups share a concern for the end users of applications. A crucial principle guiding this RFC is that **the ability for chains to provide continual service is more important than the actual upgrade burden put on the developers of these chains**. This means some extra burden for application developers is tolerable if it minimizes or substantially reduces downtime for the end user. + +### Modes of Interprocess Communication + +Tendermint has two primary mechanisms to communicate with other processes: RPC and P2P. The division marks the boundary between the internal and external components of the network: + +- The P2P layer is used in all cases that nodes (of any type) need to communicate with one another. +- The RPC interface is for any outside process that wants to communicate with a node. + +The design principle here is that **communication via RPC is to a trusted source** and thus the RPC service prioritizes inspection rather than verification. The P2P interface is the primary medium for verification. + +As an example, an in-browser light client would verify headers (and perhaps application state) via the p2p layer, and then pass along information on to the client via RPC (or potentially directly via a separate API). + +The main exceptions to this are the IBC module and relayers, which are external to the node but also require verifiable data. Breaking changes to the light client verification path mean that all neighbouring chains that are connected will no longer be able to verify state transitions and thus pass messages back and forward. + +## Proposal + +Tendermint version labels will follow the syntax of [Semantic Versions 2.0.0](https://semver.org/) with a major, minor and patch version. The version components will be interpreted according to these rules: + +For the entire cycle of a **major version** in Tendermint: + +- All blocks and state data in a blockchain can be queried. All headers can be verified even across minor version changes. Nodes can both block sync and state sync from genesis to the head of the chain. +- Nodes in a network are able to communicate and perform BFT state machine replication so long as the agreed network version is the lowest of all nodes in a network. For example, nodes using version 1.5.x and 1.2.x can operate together so long as the network version is 1.2 or lower (but still within the 1.x range). This rule essentially captures the concept of network backwards compatibility. +- Node RPC endpoints will remain compatible with existing external clients: + - New endpoints may be added, but old endpoints may not be removed. + - Old endpoints may be extended to add new request and response fields, but requests not using those fields must function as before the change. +- Migrations should be automatic. Upgrading of one node can happen asynchronously with respect to other nodes (although agreement of a network-wide upgrade must still occur synchronously via consensus). + +For the entire cycle of a **minor version** in Tendermint: + +- Public Go API's, for example in `node` or `abci` packages will not change in a way that requires any consumer (not just application developers) to modify their code. +- No breaking changes to the block protocol. This means that all block related data structures should not change in a way that breaks any of the hashes, the consensus engine or light client verification. +- Upgrades between minor versions may not result in any downtime (i.e., no migrations are required), nor require any changes to the config files to continue with the existing behavior. A minor version upgrade will require only stopping the existing process, swapping the binary, and starting the new process. + +A new **patch version** of Tendermint will only contain bug fixes and updates that impact the security and stability of Tendermint. + +These guarantees will come into effect at release 1.0. + +## Status + +Proposed + +## Consequences + +### Positive + +- Clearer communication of what versioning means to us and the effect they have on our users. + +### Negative + +- Can potentially incur greater engineering effort to uphold and follow these guarantees. + +### Neutral + +## References + +- [SemVer](https://semver.org/) +- [Tendermint Tracking Issue](https://github.com/tendermint/tendermint/issues/5680) diff --git a/cometbft/v0.38/docs/rfc/tendermint-core/rfc-015-abci++-tx-mutation.md b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-015-abci++-tx-mutation.md new file mode 100644 index 00000000..92d9ed66 --- /dev/null +++ b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-015-abci++-tx-mutation.md @@ -0,0 +1,259 @@ +# RFC 015: ABCI++ TX Mutation + +## Changelog + +- 23-Feb-2022: Initial draft (@williambanfield). +- 28-Feb-2022: Revised draft (@williambanfield). + +## Abstract + +A previous version of the ABCI++ specification detailed a mechanism for proposers to replace transactions +in the proposed block. This scheme required the proposer to construct new transactions +and mark these new transactions as replacing other removed transactions. The specification +was ambiguous as to how the replacement may be communicated to peer nodes. +This RFC discusses issues with this mechanism and possible solutions. + +## Background + +### What is the proposed change? + +A previous version of the ABCI++ specification proposed mechanisms for adding, removing, and replacing +transactions in a proposed block. To replace a transaction, the application running +`ProcessProposal` could mark a transaction as replaced by other application-supplied +transactions by returning a new transaction marked with the `ADDED` flag setting +the `new_hashes` field of the removed transaction to contain the list of transaction hashes +that replace it. In that previous specification for ABCI++, the full use of the +`new_hashes` field is left somewhat ambiguous. At present, these hashes are not +gossiped and are not eventually included in the block to signal replacement to +other nodes. The specification did indicate that the transactions specified in +the `new_hashes` field will be removed from the mempool but it's not clear how +peer nodes will learn about them. + +### What systems would be affected by adding transaction replacement? + +The 'transaction' is a central building block of a Tendermint blockchain, so adding +a mechanism for transaction replacement would require changes to many aspects of Tendermint. + +The following is a rough list of the functionality that this mechanism would affect: + +#### Transaction indexing + +Tendermint's indexer stores transactions and transaction results using the hash of the executed +transaction [as the key][tx-result-index] and the ABCI results and transaction bytes as the value. + +To allow transaction replacement, the replaced transactions would need to stored as well in the +indexer, likely as a mapping of original transaction to list of transaction hashes that replaced +the original transaction. + +#### Transaction inclusion proofs + +The result of a transaction query includes a Merkle proof of the existence of the +transaction in the block chain. This [proof is built][inclusion-proof] as a merkle tree +of the hashes of all of the transactions in the block where the queried transaction was executed. + +To allow transaction replacement, these proofs would need to be updated to prove +that a replaced transaction was included by replacement in the block. + +#### RPC-based transaction query parameters and results + +Tendermint's RPC allows clients to retrieve information about transactions via the +`/tx_search` and `/tx` RPC endpoints. + +RPC query results containing replaced transactions would need to be updated to include +information on replaced transactions, either by returning results for all of the replaced +transactions, or by including a response with just the hashes of the replaced transactions +which clients could proceed to query individually. + +#### Mempool transaction removal + +Additional logic would need to be added to the Tendermint mempool to clear out replaced +transactions after each block is executed. Tendermint currently removes executed transactions +from the mempool, so this would be a pretty straightforward change. + +## Discussion + +### What value may be added to Tendermint by introducing transaction replacement? + +Transaction replacement would would enable applications to aggregate or disaggregate transactions. + +For aggregation, a set of transactions that all related work, such as transferring +tokens between the same two accounts, could be replaced with a single transaction, +i.e. one that transfers a single sum from one account to the other. +Applications that make frequent use of aggregation may be able to achieve a higher throughput. +Aggregation would decrease the space occupied by a single client-submitted transaction in the block, allowing +more client-submitted transactions to be executed per block. + +For disaggregation, a very complex transaction could be split into multiple smaller transactions. +This may be useful if an application wishes to perform more fine-grained indexing on intermediate parts +of a multi-part transaction. + +### Drawbacks to transaction replacement + +Transaction replacement would require updating and shimming many of the places that +Tendermint records and exposes information about executed transactions. While +systems within Tendermint could be updated to account for transaction replacement, +such a system would leave new issues and rough edges. + +#### No way of guaranteeing correct replacement + +If a user issues a transaction to the network and the transaction is replaced, the +user has no guarantee that the replacement was correct. For example, suppose a set of users issue +transactions A, B, and C and they are all aggregated into a new transaction, D. +There is nothing guaranteeing that D was constructed correctly from the inputs. +The only way for users to ensure D is correct would be if D contained all of the +information of its constituent transactions, in which case, nothing is really gained by the replacement. + +#### Replacement transactions not signed by submitter + +Abstractly, Tendermint simply views transactions as a ball of bytes and therefore +should be fine with replacing one for another. However, many applications require +that transactions submitted to the chain be signed by some private key to authenticate +and authorize the transaction. Replaced transactions could not be signed by the +submitter, only by the application node. Therefore, any use of transaction replacement +could not contain authorization from the submitter and would either need to grant +application-submitted transactions power to perform application logic on behalf +of a user without their consent. + +Granting this power to application-submitted transactions would be very dangerous +and therefore might not be of much value to application developers. +Transaction replacement might only be really safe in the case of application-submitted +transactions or for transactions that require no authorization. For such transactions, +it's quite not quite clear what the utility of replacement is: the application can already +generate any transactions that it wants. The fact that such a transaction was a replacement +is not particularly relevant to participants in the chain since the application is +merely replacing its own transactions. + +#### New vector for censorship + +Depending on the implementation, transaction replacement may allow a node signal +to the rest of the chain that some transaction should no longer be considered for execution. +Honest nodes will use the replacement mechanism to signal that a transaction has been aggregated. +Malicious nodes will be granted a new vector for censoring transactions. +There is no guarantee that a replaced transactions is actually executed at all. +A malicious node could censor a transaction by simply listing it as replaced. +Honest nodes seeing the replacement would flush the transaction from their mempool +and not execute or propose it it in later blocks. + +### Transaction tracking implementations + +This section discusses possible ways to flesh out the implementation of transaction replacement. +Specifically, this section proposes a few alternative ways that Tendermint blockchains could +track and store transaction replacements. + +#### Include transaction replacements in the block + +One option to track transaction replacement is to include information on the +transaction replacement within the block. An additional structure may be added +the block of the following form: + +```proto +message Block { +... + repeated Replacement replacements = 5; +} + +message Replacement { + bytes included_tx_key = 1; + repeated bytes replaced_txs_keys = 2; +} +``` + +Applications executing `PrepareProposal` would return the list of replacements and +Tendermint would include an encoding of these replacements in the block that is gossiped +and committed. + +Tendermint's transaction indexing would include a new mapping for each replaced transaction +key to the committed transaction. +Transaction inclusion proofs would be updated to include these additional new transaction +keys in the Merkle tree and queries for transaction hashes that were replaced would return +information indicating that the transaction was replaced along with the hash of the +transaction that replaced it. + +Block validation of gossiped blocks would be updated to check that each of the +`included_txs_key` matches the hash of some transaction in the proposed block. + +Implementing the changes described in this section would allow Tendermint to gossip +and index transaction replacements as part of block propagation. These changes would +still require the application to certify that the replacements were valid. This +validation may be performed in one of two ways: + +1. **Applications optimistically trust that the proposer performed a legitimate replacement.** + +In this validation scheme, applications would not verify that the substitution +is valid during consensus and instead simply trust that the proposer is correct. +This would have the drawback of allowing a malicious proposer to remove transactions +it did not want executed. + +2. **Applications completely validate transaction replacement.** + +In this validation scheme, applications that allow replacement would check that +each listed replaced transaction was correctly reflected in the replacement transaction. +In order to perform such validation, the node would need to have the replaced transactions +locally. This could be accomplished one of a few ways: by querying the mempool, +by adding an additional p2p gossip channel for transaction replacements, or by including the replaced transactions +in the block. Replacement validation via mempool querying would require the node +to have received all of the replaced transactions in the mempool which is far from +guaranteed. Adding an additional gossip channel would make gossiping replaced transactions +a requirement for consensus to proceed, since all nodes would need to receive all replacement +messages before considering a block valid. Finally, including replaced transactions in +the block seems to obviate any benefit gained from performing a transaction replacement +since the replaced transaction and the original transactions would now both appear in the block. + +#### Application defined transaction replacement + +An additional option for allowing transaction replacement is to leave it entirely as a responsibility +of the application. The `PrepareProposal` ABCI++ call allows for applications to add +new transactions to a proposed block. Applications that wished to implement a transaction +replacement mechanism would be free to do so without the newly defined `new_hashes` field. +Applications wishing to implement transaction replacement would add the aggregated +transactions in the `PrepareProposal` response, and include one additional bookkeeping +transaction that listed all of the replacements, with a similar scheme to the `new_hashes` +field described in ABCI++. This new bookkeeping transaction could be used by the +application to determine which transactions to clear from the mempool in future calls +to `CheckTx`. + +The meaning of any transaction in the block is completely opaque to Tendermint, +so applications performing this style of replacement would not be able to have the replacement +reflected in any most of Tendermint's transaction tracking mechanisms, such as transaction indexing +and the `/tx` endpoint. + +#### Application defined Tx Keys + +Tendermint currently uses cryptographic hashes, SHA256, as a key for each transaction. +As noted in the section on systems that would require changing, this key is used +to identify the transaction in the mempool, in the indexer, and within the RPC system. + +An alternative approach to allowing `ProcessProposal` to specify a set of transaction +replacements would be instead to allow the application to specify an additional key or set +of keys for each transaction during `ProcessProposal`. This new `secondary_keys` set +would be included in the block and therefore gossiped during block propagation. +Additional RPC endpoints could be exposed to query by the application-defined keys. + +Applications wishing to implement replacement would leverage this new field by providing the +replaced transaction hashes as the `secondary_keys` and checking their validity during +`ProcessProposal`. During `RecheckTx` the application would then be responsible for +clearing out transactions that matched the `secondary_keys`. + +It is worth noting that something like this would be possible without `secondary_keys`. +An application wishing to implement a system like this one could define a replacement +transaction, as discussed in the section on application-defined transaction replacement, +and use a custom [ABCI event type][abci-event-type] to communicate that the replacement should +be indexed within Tendermint's ABCI event indexing. + +### Complexity to value-add tradeoff + +It is worth remarking that adding a system like this may introduce a decent amount +of new complexity into Tendermint. An approach that leaves much of the replacement +logic to Tendermint would require altering the core transaction indexing and querying +data. In many of the cases listed, a system for transaction replacement is possible +without explicitly defining it as part of `PrepareProposal`. Since applications +can now add transactions during `PrepareProposal` they can and should leverage this +functionality to include additional bookkeeping transactions in the block. It may +be worth encouraging applications to discover new and interesting ways to leverage this +power instead of immediately solving the problem for them. + +### References + +[inclusion-proof]: https://github.com/tendermint/tendermint/blob/0fcfaa4568cb700e27c954389c1fcd0b9e786332/types/tx.go#L67 +[tx-result-index]: https://github.com/tendermint/tendermint/blob/0fcfaa4568cb700e27c954389c1fcd0b9e786332/internal/state/indexer/tx/kv/kv.go#L90 +[abci-event-type]: https://github.com/tendermint/tendermint/blob/0fcfaa4568cb700e27c954389c1fcd0b9e786332/abci/types/types.pb.go#L3168 diff --git a/cometbft/v0.38/docs/rfc/tendermint-core/rfc-016-node-architecture.md b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-016-node-architecture.md new file mode 100644 index 00000000..29098d29 --- /dev/null +++ b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-016-node-architecture.md @@ -0,0 +1,83 @@ +# RFC 016: Node Architecture + +## Changelog + +- April 8, 2022: Initial draft (@cmwaters) +- April 15, 2022: Incorporation of feedback + +## Abstract + +The `node` package is the entry point into the Tendermint codebase, used both by the command line and programatically to create the nodes that make up a network. The package has suffered the most from the evolution of the codebase, becoming bloated as developers clipped on their bits of code here and there to get whatever feature they wanted working. + +The decisions made at the node level have the biggest impact to simplifying the protocols within them, unlocking better internal designs and making Tendermint more intuitive to use and easier to understand from the outside. Work, in minor increments, has already begun on this section of the codebase. This document exists to spark forth the necessary discourse in a few related areas that will help the team to converge on the long term makeup of the node. + +## Discussion + +The following is a list of points of discussion around the architecture of the node: + +### Dependency Tree + +The node object is currently stuffed with every component that possibly exists within Tendermint. In the constructor, all objects are built and interlaid with one another in some awkward dance. My guiding principle is that the node should only be made up of the components that it wants to have direct control of throughout its life. The node is a service which currently has the purpose of starting other services up in a particular order and stopping them all when commanded to do so. However, there are many services which are not direct dependents i.e. the mempool and evidence services should only be working when the consensus service is running. I propose to form more of a hierarchical structure of dependents which forces us to be clear about the relations that one component has to the other. More concretely, I propose the following dependency tree: + +![node dependency tree](./images/node-dependency-tree.svg) + +Many of the further discussion topics circle back to this representation of the node. + +It's also important to distinguish two dimensions which may require different characteristics of the architecture. There is the starting and stopping of services and their general lifecycle management. What is the correct order of operations to starting a node for example. Then there is the question of the needs of the service during actual operation. Then there is the question of what resources each service needs access to during its operation. Some need to publish events, others need access to data stores, and so forth. + +An alternative model and one that perhaps better suits the latter of these dimensions is the notion of an internal message passing system. Either the events bus or p2p layer could serve as a viable transport. This would essentially allow all services to communicate with any other service and could perhaps provide a solution to the coordination problem (presented below) without a centralized coordinator. The other main advantage is that such a system would be more robust to disruptions and changes to the code which may make a hierarchical structure quickly outdated and suboptimal. The addition of message routing is an added complexity to implement, will increase the degree of asynchronicity in the system and may make it harder to debug problems that are across multiple services. + +### Coordination of State Advancing Mechanisms + +Advancement of state in Tendermint is simply defined in heights: If the node is at height n, how does it get to height n + 1 and so on. Based on this definition we have three components that help a node to advance in height: consensus, statesync and blocksync. The way these components behave currently is very tightly coupled to one another with references passed back and forth. My guiding principle is that each of these should be able to operate completely independently of each other, e.g. a node should be able to run solely blocksync indefinitely. There have been several ideas suggested towards improving this flow. I've been leaning strongly towards a centralized system, whereby an orchestrator (in this case the node) decides what services to start and stop. +In a decentralized message passing system, individual services make their decision based upon a "global" shared state i.e. if my height is less that 10 below the average peer height, I as consensus, should stop (knowing that blocksync has the same condition for starting). As the example illustrates, each mechanism will still need to be aware of the presence of other mechanisms. + +Both centralized and decentralized systems rely on the communication of the nodes current height and a judgement on the height of the head of the chain. The latter, working out the head of the chain, is quite a difficult challenge as their is nothing preventing the node from acting maliciously and providing a different height. Currently both blocksync, consensus (and to a certain degree statesync), have parallel systems where peers communicate their height. This could be streamlined with the consensus (or even the p2p layer), broadcasting peer heights and either the node or the other state advancing mechanisms acting accordingly. + +Currently, when a node starts, it turns on every service that it is attached to. This means that while a node is syncing up by requesting blocks, it is also receiving transactions and votes, as well as snapshot and block requests. This is a needless use of bandwidth. An implementation of an orchestrator, regardless of whether the system is heirachical or not, should look to be able to open and close channels dynamically and effectively broadcast which services it is running. Integrating this with service discovery may also lead to a better serivce to peers. + +The orchestrator allows for some deal of variablity in how a node is constructed. Does it just run blocksync, shadowing the head of the chain and be highly available for querying. Does it rely on state sync at all? An important question that arises from this dynamicism is we ideally want to encourage nodes to provide as much of their resources as possible so that their is a healthy amount of providers to consumers. Do we make all services compulsory or allow for them to be disabled? Arguably it's possible that a user forks the codebase and rips out the blocksync code because they want to reduce bandwidth so this is more a question of how easy do we want to make this for users. + +### Block Executor + +The block executor is an important component that is currently used by both consensus and blocksync to execute transactions and update application state. Principally, I think it should be the only component that can write (and possibly even read) the block and state stores, and we should clean up other direct dependencies on the storage engine if we can. This would mean: + +- The reactors Consensus, BlockSync and StateSync should all import the executor for advancing state ie. `ApplyBlock` and `BootstrapState`. +- Pruning should also be a concern of the block executor as well as `FinalizeBlock` and `Commit`. This can simplify consensus to focus just on the consensus part. + +### The Interprocess communication systems: RPC, P2P, ABCI, and Events + +The schematic supplied above shows the relations between the different services, the node, the block executor, and the storage layer. Represented as colored dots are the components responsible for different roles of interprocess communication (IPC). These components permeate throughout the code base, seeping into most services. What can provide powerful functionality on one hand can also become a twisted vine, creating messy corner cases and convoluting the protocols themselves. A lot of the thinking around +how we want our IPC systens to function has been summarised in this [RFC](./rfc-002-ipc-ecosystem.md). In this section, I'd like to focus the reader on the relation between the IPC and the node structure. An issue that has frequently risen is that the RPC has control of the components where it strikes me as being more logical for the component to dictate the information that is emitted/available and the knobs it wishes to expose. The RPC is also inextricably tied to the node instance and has situations where it is passed pointers directly to the storage engine and other components. + +I am currently convinced of the approach that the p2p layer takes and would like to see other IPC components follow suit. This would mean that the RPC and events system would be constructed in the node yet would pass the adequate methods to register endpoints and topics to the sub components. For example, + +```go +// Methods from the RPC and event bus that would be passed into the constructor of components like "consensus" +// NOTE: This is a hypothetical construction to convey the idea. An actual implementation may differ. +func RegisterRoute(path string, handler func(http.ResponseWriter, *http.Request)) + +func RegisterTopic(name string) EventPublisher + +type EventPublisher func (context.Context, types.EventData, []abci.Event) +``` + +This would give the components control to the information they want to expose and keep all relevant logic within that package. It accomodates more to a dynamic system where services can switch on and off. Each component would also receive access to the logger and metrics system for introspection and debuggability. + +#### IPC Rubric + +I'd like to aim to reach a state where we as a team have either an implicit or explicit rubric which can determine, in the event of some new need to communicate information, what tool it should use for doing this. In the case of inter node communication, this is obviously the p2p stack (with perhaps the exception of the light client). Metrics and logging also have clear usage patterns. RPC and the events system are less clear. The RPC is used for debugging data and fine tuned operator control as it is for general public querying and transaction submission. The RPC is also known to have been plumbed back into the application for historical queries. The events system, similarly, is used for consuming transaction events as it is for the testing of consensus state transitions. + +Principally, I think we should look to change our language away from what the actual transport is and more towards what it's being used for and to whom. We call it a peer to peer layer and not the underlying tcp connection. In the same way, we should look to split RPC into an operator interface (RPC Internal), a public interface (RPC External) and a bidirectional ABCI. + +### Seperation of consumers and suppliers + +When a service such as blocksync is turned on, it automatically begins requesting blocks to verify and apply them as it also tries to serve them to other peers catching up. We should look to distinguish these two aspects: supplying of information and consuming of information in many of these components. More concretely, I'd suggest: + +- The blocksync and statesync service, i.e. supplying information for those trying to catch up should only start running once a node has caught up i.e. after running the blocksync and/or state sync *processes* +- The blocksync and state sync processes have defined termination clauses that inform the orchestrator when they are done and where they finished. + - One way of achieving this would be that every process both passes and returns the `State` object + - In some cases, a node may specify that it wants to run blocksync indefinitely. +- The mempool should also indicate whether it wants to receive transactions or to send them only (one-directional mempool) +- Similarly, the light client itself only requests information whereas the light client service (currently part of state sync) can do both. +- This distinction needs to be communicated in the p2p layer handshake itself but should also be changeable over the lifespan of the connection. diff --git a/cometbft/v0.38/docs/rfc/tendermint-core/rfc-017-abci++-vote-extension-propag.md b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-017-abci++-vote-extension-propag.md new file mode 100644 index 00000000..15d08f7b --- /dev/null +++ b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-017-abci++-vote-extension-propag.md @@ -0,0 +1,571 @@ +# RFC 017: ABCI++ Vote Extension Propagation + +## Changelog + +- 11-Apr-2022: Initial draft (@sergio-mena). +- 15-Apr-2022: Addressed initial comments. First complete version (@sergio-mena). +- 09-May-2022: Addressed all outstanding comments. + +## Abstract + +According to the +[ABCI++ specification](https://github.com/tendermint/tendermint/blob/4743a7ad0/spec/abci%2B%2B/README.md) +(as of 11-Apr-2022), a validator MUST provide a signed vote extension for each non-`nil` precommit vote +of height *h* that it uses to propose a block in height *h+1*. When a validator is up to +date, this is easy to do, but when a validator needs to catch up this is far from trivial as this data +cannot be retrieved from the blockchain. + +This RFC presents and compares the different options to address this problem, which have been proposed +in several discussions by the Tendermint Core team. + +## Document Structure + +The RFC is structured as follows. In the [Background](#background) section, +subsections [Problem Description](#problem-description) and [Cases to Address](#cases-to-address) +explain the problem at hand from a high level perspective, i.e., abstracting away from the current +Tendermint implementation. In contrast, subsection +[Current Catch-up Mechanisms](#current-catch-up-mechanisms) delves into the details of the current +Tendermint code. + +In the [Discussion](#discussion) section, subsection [Solutions Proposed](#solutions-proposed) is also +worded abstracting away from implementation details, whilst subsections +[Feasibility of the Proposed Solutions](#feasibility-of-the-proposed-solutions) and +[Current Limitations and Possible Implementations](#current-limitations-and-possible-implementations) +analize the viability of one of the proposed solutions in the context of Tendermint's architecture +based on reactors. Finally, [Formalization Work](#formalization-work) briefly discusses the work +still needed demonstrate the correctness of the chosen solution. + +The high level subsections are aimed at readers who are familiar with consensus algorithms, in +particular with the one described in the Tendermint (white paper), but who are not necessarily +acquainted with the details of the Tendermint codebase. The other subsections, which go into +implementation details, are best understood by engineers with deep knowledge of the implementation of +Tendermint's blocksync and consensus reactors. + +## Background + +### Basic Definitions + +This document assumes that all validators have equal voting power for the sake of simplicity. This is done +without loss of generality. + +There are two types of votes in Tendermint: *prevotes* and *precommits*. Votes can be `nil` or refer to +a proposed block. This RFC focuses on precommits, +also known as *precommit votes*. In this document we sometimes call them simply *votes*. + +Validators send precommit votes to their peer nodes in *precommit messages*. According to the +[ABCI++ specification](https://github.com/tendermint/tendermint/blob/4743a7ad0/spec/abci%2B%2B/README.md), +a precommit message MUST also contain a *vote extension*. +This mandatory vote extension can be empty, but MUST be signed with the same key as the precommit +vote (i.e., the sending validator's). +Nevertheless, the vote extension is signed independently from the vote, so a vote can be separated from +its extension. +The reason for vote extensions to be mandatory in precommit messages is that, otherwise, a (malicious) +node can omit a vote extension while still providing/forwarding/sending the corresponding precommit vote. + +The validator set at height *h* is denoted *valseth*. A *commit* for height *h* consists of more +than *2nh/3* precommit votes voting for a block *b*, where *nh* denotes the size of +*valseth*. A commit does not contain `nil` precommit votes, and all votes in it refer to the +same block. An *extended commit* is a *commit* where every precommit vote has its respective vote extension +attached. + +### Problem Description + +In the version of [ABCI](https://github.com/tendermint/spec/blob/4fb99af/spec/abci/README.md) present up to +Tendermint v0.35, for any height *h*, a validator *v* MUST have the decided block *b* and a commit for +height *h* in order to decide at height *h*. Then, *v* just needs a commit for height *h* to propose at +height *h+1*, in the rounds of *h+1* where *v* is a proposer. + +In [ABCI++](https://github.com/tendermint/tendermint/blob/4743a7ad0/spec/abci%2B%2B/README.md), +the information that a validator *v* MUST have to be able to decide in *h* does not change with +respect to pre-existing ABCI: the decided block *b* and a commit for *h*. +In contrast, for proposing in *h+1*, a commit for *h* is not enough: *v* MUST now have an extended +commit. + +When a validator takes an active part in consensus at height *h*, it has all the data it needs in memory, +in its consensus state, to decide on *h* and propose in *h+1*. Things are not so easy in the cases when +*v* cannot take part in consensus because it is late (e.g., it falls behind, it crashes +and recovers, or it just starts after the others). If *v* does not take part, it cannot actively +gather precommit messages (which include vote extensions) in order to decide. +Before ABCI++, this was not a problem: full nodes are supposed to persist past blocks in the block store, +so other nodes would realise that *v* is late and send it the missing decided block at height *h* and +the corresponding commit (kept in block *h+1*) so that *v* can catch up. +However, we cannot apply this catch-up technique for ABCI++, as the vote extensions, which are part +of the needed *extended commit* are not part of the blockchain. + +### Cases to Address + +Before we tackle the description of the possible cases we need to address, let us describe the following +incremental improvement to the ABCI++ logic. Upon decision, a full node persists (e.g., in the block +store) the extended commit that allowed the node to decide. For the moment, let us assume the node only +needs to keep its *most recent* extended commit, and MAY remove any older extended commits from persistent +storage. +This improvement is so obvious that all solutions described in the [Discussion](#discussion) section use +it as a building block. Moreover, it completely addresses by itself some of the cases described in this +subsection. + +We now describe the cases (i.e. possible *runs* of the system) that have been raised in different +discussions and need to be addressed. They are (roughly) ordered from easiest to hardest to deal with. + +- **(a)** *Happy path: all validators advance together, no crash*. + + This case is included for completeness. All validators have taken part in height *h*. + Even if some of them did not manage to send a precommit message for the decided block, they all + receive enough precommit messages to be able to decide. As vote extensions are mandatory in + precommit messages, every validator *v* trivially has all the information, namely the decided block + and the extended commit, needed to propose in height *h+1* for the rounds in which *v* is the + proposer. + + No problem to solve here. + +- **(b)** *All validators advance together, then all crash at the same height*. + + This case has been raised in some discussions, the main concern being whether the vote extensions + for the previous height would be lost across the network. With the improvement described above, + namely persisting the latest extended commit at decision time, this case is solved. + When a crashed validator recovers, it recovers the last extended commit from persistent storage + and handshakes with the Application. + If need be, it also reconstructs messages for the unfinished height + (including all precommits received) from the WAL. + Then, the validator can resume where it was at the time of the crash. Thus, as extensions are + persisted, either in the WAL (in the form of received precommit messages), or in the latest + extended commit, the only way that vote extensions needed to start the next height could be lost + forever would be if all validators crashed and never recovered (e.g. disk corruption). + Since a *correct* node MUST eventually recover, this violates Tendermint's assumption of more than + *2nh/3* correct validators for every height *h*. + + No problem to solve here. + +- **(c)** *Lagging majority*. + + Let us assume the validator set does not change between *h* and *h+1*. + It is not possible by the nature of the Tendermint algorithm, which requires more + than *2nh/3* precommit votes for some round of height *h* in order to make progress. + So, only up to *nh/3* validators can lag behind. + + On the other hand, for the case where there are changes to the validator set between *h* and + *h+1* please see case (d) below, where the extreme case is discussed. + +- **(d)** *Validator set changes completely between* h *and* h+1. + + If sets *valseth* and *valseth+1* are disjoint, + more than *2nh/3* of validators in height *h* should + have actively participated in conensus in *h*. So, as of height *h*, only a minority of validators + in *h* can be lagging behind, although they could all lag behind from *h+1* on, as they are no + longer validators, only full nodes. This situation falls under the assumptions of case (h) below. + + As for validators in *valseth+1*, as they were not validators as of height *h*, they + could all be lagging behind by that time. However, by the time *h* finishes and *h+1* begins, the + chain will halt until more than *2nh+1/3* of them have caught up and started consensus + at height *h+1*. If set *valseth+1* does not change in *h+2* and subsequent + heights, only up to *nh+1/3* validators will be able to lag behind. Thus, we have + converted this case into case (h) below. + +- **(e)** *Enough validators crash to block the rest*. + + In this case, blockchain progress halts, i.e. surviving full nodes keep increasing rounds + indefinitely, until some of the crashed validators are able to recover. + Those validators that recover first will handshake with the Application and recover at the height + they crashed, which is still the same the nodes that did not crash are stuck in, so they don't need + to catch up. + Further, they had persisted the extended commit for the previous height. Nothing to solve. + + For those validators recovering later, we are in case (h) below. + +- **(f)** *Some validators crash, but not enough to block progress*. + + When the correct processes that crashed recover, they handshake with the Application and resume at + the height they were at when they crashed. As the blockchain did not stop making progress, the + recovered processes are likely to have fallen behind with respect to the progressing majority. + + At this point, the recovered processes are in case (h) below. + +- **(g)** *A new full node starts*. + + The reasoning here also applies to the case when more than one full node are starting. + When the full node starts from scratch, it has no state (its current height is 0). Ignoring + statesync for the time being, the node just needs to catch up by applying past blocks one by one + (after verifying them). + + Thus, the node is in case (h) below. + +- **(h)** *Advancing majority, lagging minority* + + In this case, some nodes are late. More precisely, at the present time, a set of full nodes, + denoted *Lhp*, are falling behind + (e.g., temporary disconnection or network partition, memory thrashing, crashes, new nodes) + an arbitrary + number of heights: + between *hs* and *hp*, where *hs < hp*, and + *hp* is the highest height + any correct full node has reached so far. + + The correct full nodes that reached *hp* were able to decide for *hp-1*. + Therefore, less than *nhp-1/3* validators of *hp-1* can be part + of *Lhp*, since enough up-to-date validators needed to actively participate + in consensus for *hp-1*. + + Since, at the present time, + no node in *Lhp* took part in any consensus between + *hs* and *hp-1*, + the reasoning above can be extended to validator set changes between *hs* and + *hp-1*. This results in the following restriction on the full nodes that can be part of *Lhp*. + + - ∀ *h*, where *hs ≤ h < hp*, + | *valseth* ∩ *Lhp* | *< nh/3* + + If this property does not hold for a particular height *h*, where + *hs ≤ h < hp*, Tendermint could not have progressed beyond *h* and + therefore no full node could have reached *hp* (a contradiction). + + These lagging nodes in *Lhp* need to catch up. They have to obtain the + information needed to make + progress from other nodes. For each height *h* between *hs* and *hp-2*, + this includes the decided block for *h*, and the + precommit votes also for *deciding h* (which can be extracted from the block at height *h+1*). + + At a given height *hc* (where possibly *hc << hp*), + a full node in *Lhp* will consider itself *caught up*, based on the + (maybe out of date) information it is getting from its peers. Then, the node needs to be ready to + propose at height *hc+1*, which requires having received the vote extensions for + *hc*. + As the vote extensions are *not* stored in the blocks, and it is difficult to have strong + guarantees on *when* a late node considers itself caught up, providing the late node with the right + vote extensions for the right height poses a problem. + +At this point, we have described and compared all cases raised in discussions leading up to this +RFC. The list above aims at being exhaustive. The analysis of each case included above makes all of +them converge into case (h). + +### Current Catch-up Mechanisms + +We now briefly describe the current catch-up mechanisms in the reactors concerned in Tendermint. + +#### Statesync + +Full nodes optionally run statesync just after starting, when they start from scratch. +If statesync succeeds, an Application snapshot is installed, and Tendermint jumps from height 0 directly +to the height the Application snapshop represents, without applying the block of any previous height. +Some light blocks are received and stored in the block store for running light-client verification of +all the skipped blocks. Light blocks are incomplete blocks, typically containing the header and the +canonical commit but, e.g., no transactions. They are stored in the block store as "signed headers". + +The statesync reactor is not really relevant for solving the problem discussed in this RFC. We will +nevertheless mention it when needed; in particular, to understand some corner cases. + +#### Blocksync + +The blocksync reactor kicks in after start up or recovery (and, optionally, after statesync is done) +and sends the following messages to its peers: + +- `StatusRequest` to query the height its peers are currently at, and +- `BlockRequest`, asking for blocks of heights the local node is missing. + +Using `BlockResponse` messages received from peers, the blocksync reactor validates each received +block using the block of the following height, saves the block in the block store, and sends the +block to the Application for execution. + +If blocksync has validated and applied the block for the height *previous* to the highest seen in +a `StatusResponse` message, or if no progress has been made after a timeout, the node considers +itself as caught up and switches to the consensus reactor. + +#### Consensus Reactor + +The consensus reactor runs the full Tendermint algorithm. For a validator this means it has to +propose blocks, and send/receive prevote/precommit messages, as mandated by Tendermint, before it can +decide and move on to the next height. + +If a full node that is running the consensus reactor falls behind at height *h*, when a peer node +realises this it will retrieve the canonical commit of *h+1* from the block store, and *convert* +it into a set of precommit votes and will send those to the late node. + +## Discussion + +### Solutions Proposed + +These are the solutions proposed in discussions leading up to this RFC. + +- **Solution 0.** *Vote extensions are made **best effort** in the specification*. + + This is the simplest solution, considered as a way to provide vote extensions in a simple enough + way so that it can be part of v0.36. + It consists in changing the specification so as to not *require* that precommit votes used upon + `PrepareProposal` contain their corresponding vote extensions. In other words, we render vote + extensions optional. + There are strong implications stemming from such a relaxation of the original specification. + + - As a vote extension is signed *separately* from the vote it is extending, an intermediate node + can now remove (i.e., censor) vote extensions from precommit messages at will. + - Further, there is no point anymore in the spec requiring the Application to accept a vote extension + passed via `VerifyVoteExtension` to consider a precommit message valid in its entirety. Remember + this behavior of `VerifyVoteExtension` is adding a constraint to Tendermint's conditions for + liveness. + In this situation, it is better and simpler to just drop the vote extension rejected by the + Application via `VerifyVoteExtension`, but still consider the precommit vote itself valid as long + as its signature verifies. + +- **Solution 1.** *Include vote extensions in the blockchain*. + + Another obvious solution, which has somehow been considered in the past, is to include the vote + extensions and their signatures in the blockchain. + The blockchain would thus include the extended commit, rather than a regular commit, as the structure + to be canonicalized in the next block. + With this solution, the current mechanisms implemented both in the blocksync and consensus reactors + would still be correct, as all the information a node needs to catch up, and to start proposing when + it considers itself as caught-up, can now be recovered from past blocks saved in the block store. + + This solution has two main drawbacks. + + - As the block format must change, upgrading a chain requires a hard fork. Furthermore, + all existing light client implementations will stop working until they are upgraded to deal with + the new format (e.g., how certain hashes calculated and/or how certain signatures are checked). + For instance, let us consider IBC, which relies on light clients. An IBC connection between + two chains will be broken if only one chain upgrades. + - The extra information (i.e., the vote extensions) that is now kept in the blockchain is not really + needed *at every height* for a late node to catch up. + - This information is only needed to be able to *propose* at the height the validator considers + itself as caught-up. If a validator is indeed late for height *h*, it is useless (although + correct) for it to call `PrepareProposal`, or `ExtendVote`, since the block is already decided. + - Moreover, some use cases require pretty sizeable vote extensions, which would result in an + important waste of space in the blockchain. + +- **Solution 2.** *Skip* propose *step in Tendermint algorithm*. + + This solution consists in modifying the Tendermint algorithm to skip the *send proposal* step in + heights where the node does not have the required vote extensions to populate the call to + `PrepareProposal`. The main idea behind this is that it should only happen when the validator is late + and, therefore, up-to-date validators have already proposed (and decided) for that height. + A small variation of this solution is, rather than skipping the *send proposal* step, the validator + sends a special *empty* or *bottom* (⊥) proposal to signal other nodes that it is not ready to propose + at (any round of) the current height. + + The appeal of this solution is its simplicity. A possible implementation does not need to extend + the data structures, or change the current catch-up mechanisms implemented in the blocksync or + in the consensus reactor. When we lack the needed information (vote extensions), we simply rely + on another correct validator to propose a valid block in other rounds of the current height. + + However, this solution can be attacked by a byzantine node in the network in the following way. + Let us consider the following scenario: + + - all validators in *valseth* send out precommit messages, with vote extensions, + for height *h*, round 0, roughly at the same time, + - all those precommit messages contain non-`nil` precommit votes, which vote for block *b* + - all those precommit messages sent in height *h*, round 0, and all messages sent in + height *h*, round *r > 0* get delayed indefinitely, so, + - all validators in *valseth* keep waiting for enough precommit + messages for height *h*, round 0, needed for deciding in height *h* + - an intermediate (malicious) full node *m* manages to receive block *b*, and gather more than + *2nh/3* precommit messages for height *h*, round 0, + - one way or another, the solution should have either (a) a mechanism for a full node to *tell* + another full node it is late, or (b) a mechanism for a full node to conclude it is late based + on other full nodes' messages; any of these mechanisms should, at the very least, + require the late node receiving the decided block and a commit (not necessarily an extended + commit) for *h*, + - node *m* uses the gathered precommit messages to build a commit for height *h*, round 0, + - in order to convince full nodes that they are late, node *m* either (a) *tells* them they + are late, or (b) shows them it (i.e. *m*) is ahead, by sending them block *b*, along with the + commit for height *h*, round 0, + - all full nodes conclude they are late from *m*'s behavior, and use block *b* and the commit for + height *h*, round 0, to decide on height *h*, and proceed to height *h+1*. + + At this point, *all* full nodes, including all validators in *valseth+1*, have advanced + to height *h+1* believing they are late, and so, expecting the *hypothetical* leading majority of + validators in *valseth+1* to propose for *h+1*. As a result, the blockhain + grinds to a halt. + A (rather complex) ad-hoc mechanism would need to be carried out by node operators to roll + back all validators to the precommit step of height *h*, round *r*, so that they can regenerate + vote extensions (remember vote extensions are non-deterministic) and continue execution. + +- **Solution 3.** *Require extended commits to be available at switching time*. + + This one is more involved than all previous solutions, and builds on an idea present in Solution 2: + vote extensions are actually not needed for Tendermint to make progress as long as the + validator is *certain* it is late. + + We define two modes. The first is denoted *catch-up mode*, and Tendermint only calls + `FinalizeBlock` for each height when in this mode. The second is denoted *consensus mode*, in + which the validator considers itself up to date and fully participates in consensus and calls + `PrepareProposal`/`ProcessProposal`, `ExtendVote`, and `VerifyVoteExtension`, before calling + `FinalizeBlock`. + + The catch-up mode does not need vote extension information to make progress, as all it needs is the + decided block at each height to call `FinalizeBlock` and keep the state-machine replication making + progress. The consensus mode, on the other hand, does need vote extension information when + starting every height. + + Validators are in consensus mode by default. When a validator in consensus mode falls behind + for whatever reason, e.g. cases (b), (d), (e), (f), (g), or (h) above, we introduce the following + key safety property: + + - for every height *hp*, a full node *f* in *hp* refuses to switch to catch-up + mode **until** there exists a height *h'* such that: + - *p* has received and (light-client) verified the blocks of + all heights *h*, where *hp ≤ h ≤ h'* + - it has received an extended commit for *h'* and has verified: + - the precommit vote signatures in the extended commit + - the vote extension signatures in the extended commit: each is signed with the same + key as the precommit vote it extends + + If the condition above holds for *hp*, namely receiving a valid sequence of blocks in + the *f*'s future, and an extended commit corresponding to the last block in the sequence, then + node *f*: + + - switches to catch-up mode, + - applies all blocks between *hp* and *h'* (calling `FinalizeBlock` only), and + - switches back to consensus mode using the extended commit for *h'* to propose in the rounds of + *h' + 1* where it is the proposer. + + This mechanism, together with the invariant it uses, ensures that the node cannot be attacked by + being fed a block without extensions to make it believe it is late, in a similar way as explained + for Solution 2. + +### Feasibility of the Proposed Solutions + +Solution 0, besides the drawbacks described in the previous section, provides guarantees that are +weaker than the rest. The Application does not have the assurance that more than *2nh/3* vote +extensions will *always* be available when calling `PrepareProposal` at height *h+1*. +This level of guarantees is probably not strong enough for vote extensions to be useful for some +important use cases that motivated them in the first place, e.g., encrypted mempool transactions. + +Solution 1, while being simple in that the changes needed in the current Tendermint codebase would +be rather small, is changing the block format, and would therefore require all blockchains using +Tendermint v0.35 or earlier to hard-fork when upgrading to v0.36. + +Since Solution 2 can be attacked, one might prefer Solution 3, even if it is more involved +to implement. Further, we must elaborate on how we can turn Solution 3, described in abstract +terms in the previous section, into a concrete implementation compatible with the current +Tendermint codebase. + +### Current Limitations and Possible Implementations + +The main limitations affecting the current version of Tendermint are the following. + +- The current version of the blocksync reactor does not use the full + [light client verification](https://github.com/tendermint/tendermint/blob/4743a7ad0/spec/light-client/README.md) + algorithm to validate blocks coming from other peers. +- The code being structured into the blocksync and consensus reactors, only switching from the + blocksync reactor to the consensus reactor is supported; switching in the opposite direction is + not supported. Alternatively, the consensus reactor could have a mechanism allowing a late node + to catch up by skipping calls to `PrepareProposal`/`ProcessProposal`, and + `ExtendVote`/`VerifyVoteExtension` and only calling `FinalizeBlock` for each height. + Such a mechanism does not exist at the time of writing this RFC. + +The blocksync reactor featuring light client verification is being actively worked on (tentatively +for v0.37). So it is best if this RFC does not try to delve into that problem, but just makes sure +its outcomes are compatible with that effort. + +In subsection [Cases to Address](#cases-to-address), we concluded that we can focus on +solving case (h) in theoretical terms. +However, as the current Tendermint version does not yet support switching back to blocksync once a +node has switched to consensus, we need to split case (h) into two cases. When a full node needs to +catch up... + +- **(h.1)** ... it has not switched yet from the blocksync reactor to the consensus reactor, or + +- **(h.2)** ... it has already switched to the consensus reactor. + +This is important in order to discuss the different possible implementations. + +#### Base Implementation: Persist and Propagate Extended Commit History + +In order to circumvent the fact that we cannot switch from the consensus reactor back to blocksync, +rather than just keeping the few most recent extended commits, nodes will need to keep +and gossip a backlog of extended commits so that the consensus reactor can still propose and decide +in out-of-date heights (even if those proposals will be useless). + +The base implementation - for which an experimental patch exists - consists in the conservative +approach of persisting in the block store *all* extended commits for which we have also stored +the full block. Currently, when statesync is run at startup, it saves light blocks. +This base implementation does not seek +to receive or persist extended commits for those light blocks as they would not be of any use. + +Then, we modify the blocksync reactor so that peers *always* send requested full blocks together +with the corresponding extended commit in the `BlockResponse` messages. This guarantees that the +block store being reconstructed by blocksync has the same information as that of peers that are +up to date (at least starting from the latest snapshot applied by statesync before starting blocksync). +Thus, blocksync has all the data it requires to switch to the consensus reactor, as long as one of +the following exit conditions are met: + +- The node is still at height 0 (where no commit or extended commit is needed) +- The node has processed at least 1 block in blocksync + +The second condition is needed in case the node has installed an Application snapshot during statesync. +If that is the case, at the time blocksync starts, the block store only has the data statesync has saved: +light blocks, and no extended commits. +Hence we need to blocksync at least one block from another node, which will be sent with its corresponding extended commit, before we can switch to consensus. + +As a side note, a chain might be started at a height *hi > 0*, all other heights +*h < hi* being non-existent. In this case, the chain is still considered to be at height 0 before +block *hi* is applied, so the first condition above allows the node to switch to consensus even +if blocksync has not processed any block (which is always the case if all nodes are starting from scratch). + +When a validator falls behind while having already switched to the consensus reactor, a peer node can +simply retrieve the extended commit for the required height from the block store and reconstruct a set of +precommit votes together with their extensions and send them in the form of precommit messages to the +validator falling behind, regardless of whether the peer node holds the extended commit because it +actually participated in that consensus and thus received the precommit messages, or it received the extended commit via a `BlockResponse` message while running blocksync. + +This solution requires a few changes to the consensus reactor: + +- upon saving the block for a given height in the block store at decision time, save the + corresponding extended commit as well +- in the catch-up mechanism, when a node realizes that another peer is more than 2 heights + behind, it uses the extended commit (rather than the canoncial commit as done previously) to + reconstruct the precommit votes with their corresponding extensions + +The changes to the blocksync reactor are more substantial: + +- the `BlockResponse` message is extended to include the extended commit of the same height as + the block included in the response (just as they are stored in the block store) +- structure `bpRequester` is likewise extended to hold the received extended commits coming in + `BlockResponse` messages +- method `PeekTwoBlocks` is modified to also return the extended commit corresponding to the first block +- when successfully verifying a received block, the reactor saves its corresponding extended commit in + the block store + +The two main drawbacks of this base implementation are: + +- the increased size taken by the block store, in particular with big extensions +- the increased bandwith taken by the new format of `BlockResponse` + +#### Possible Optimization: Pruning the Extended Commit History + +If we cannot switch from the consensus reactor back to the blocksync reactor we cannot prune the extended commit backlog in the block store without sacrificing the implementation's correctness. The asynchronous +nature of our distributed system model allows a process to fall behing an arbitrary number of +heights, and thus all extended commits need to be kept *just in case* a node that late had +previously switched to the consensus reactor. + +However, there is a possibility to optimize the base implementation. Every time we enter a new height, +we could prune from the block store all extended commits that are more than *d* heights in the past. +Then, we need to handle two new situations, roughly equivalent to cases (h.1) and (h.2) described above. + +- (h.1) A node starts from scratch or recovers after a crash. In thisy case, we need to modify the + blocksync reactor's base implementation. + - when receiving a `BlockResponse` message, it MUST accept that the extended commit set to `nil`, + - when sending a `BlockResponse` message, if the block store contains the extended commit for that + height, it MUST set it in the message, otherwise it sets it to `nil`, + - the exit conditions used for the base implementation are no longer valid; the only reliable exit + condition now consists in making sure that the last block processed by blocksync was received with + the corresponding commit, and not `nil`; this extended commit will allow the node to switch from + the blocksync reactor to the consensus reactor and immediately act as a proposer if required. +- (h.2) A node already running the consensus reactor falls behind beyond *d* heights. In principle, + the node will be stuck forever as no other node can provide the vote extensions it needs to make + progress (they all have pruned the corresponding extended commit). + However we can manually have the node crash and recover as a workaround. This effectively converts + this case into (h.1). + +### Formalization Work + +A formalization work to show or prove the correctness of the different use cases and solutions +presented here (and any other that may be found) needs to be carried out. +A question that needs a precise answer is how many extended commits (one?, two?) a node needs +to keep in persistent memory when implementing Solution 3 described above without Tendermint's +current limitations. +Another important invariant we need to prove formally is that the set of vote extensions +required to make progress will always be held somewhere in the network. + +## References + +- [ABCI++ specification](https://github.com/tendermint/tendermint/blob/4743a7ad0/spec/abci%2B%2B/README.md) +- [ABCI as of v0.35](https://github.com/tendermint/spec/blob/4fb99af/spec/abci/README.md) +- [Vote extensions issue](https://github.com/tendermint/tendermint/issues/8174) +- [Light client verification](https://github.com/tendermint/tendermint/blob/4743a7ad0/spec/light-client/README.md) diff --git a/cometbft/v0.38/docs/rfc/tendermint-core/rfc-018-bls-agg-exploration.md b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-018-bls-agg-exploration.md new file mode 100644 index 00000000..6de7510a --- /dev/null +++ b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-018-bls-agg-exploration.md @@ -0,0 +1,551 @@ +# RFC 018: BLS Signature Aggregation Exploration + +## Changelog + +- 01-April-2022: Initial draft (@williambanfield). +- 15-April-2022: Draft complete (@williambanfield). + +## Abstract + +## Background + +### Glossary + +The terms that are attached to these types of cryptographic signing systems +become confusing quickly. Different sources appear to use slightly different +meanings of each term and this can certainly add to the confusion. Below is +a brief glossary that may be helpful in understanding the discussion that follows. + +- **Short Signature**: A signature that does not vary in length with the +number of signers. +- **Multi-Signature**: A signature generated over a single message +where, given the message and signature, a verifier is able to determine that +all parties signed the message. May be short or may vary with the number of signers. +- **Aggregated Signature**: A _short_ signature generated over messages with +possibly different content where, given the messages and signature, a verifier +should be able to determine that all the parties signed the designated messages. +- **Threshold Signature**: A _short_ signature generated from multiple signers +where, given a message and the signature, a verifier is able to determine that +a large enough share of the parties signed the message. The identities of the +parties that contributed to the signature are not revealed. +- **BLS Signature**: An elliptic-curve pairing-based signature system that +has some nice properties for short multi-signatures. May stand for +_Boneh-Lynn-Schacham_ or _Barreto-Lynn-Scott_ depending on the context. A +BLS signature is type of signature scheme that is distinct from other forms +of elliptic-curve signatures such as ECDSA and EdDSA. +- **Interactive**: Cryptographic scheme where parties need to perform one or +more request-response cycles to produce the cryptographic material. For +example, an interactive signature scheme may require the signer and the +verifier to cooperate to create and/or verify the signature, rather than a +signature being created ahead of time. +- **Non-interactive**: Cryptographic scheme where parties do not need to +perform any request-response cycles to produce the cryptographic material. + +### Brief notes on pairing-based elliptic-curve cryptography + +Pairing-based elliptic-curve cryptography is quite complex and relies on several +types of high-level math. Cryptography, in general, relies on being able to find +problems with an asymmetry between the difficulty of calculating the solution +and verifying that a given solution is correct. + +Pairing-based cryptography works by operating on mathematical functions that +satisfy the property of **bilinear mapping**. This property is satisfied for +functions `e` with values `P`, `Q`, `R` and `S` where `e(P, Q + R) = e(P, Q) * e(P, R)` +and `e(P + S, Q) = e(P, Q) * e(S, Q)`. The most familiar example of this is +exponentiation. Written in common notation, `g^P*(Q+R) = g^(P*Q) * g^(P*R)` for +some value `g`. + +Pairing-based elliptic-curve cryptography creates a bilinear mapping using +elliptic curves over a finite field. With some original curve, you can define two groups, +`G1` and `G2` which are points of the original curve _modulo_ different values. +Finally, you define a third group `Gt`, where points from `G1` and `G2` satisfy +the property of bilinearity with `Gt`. In this scheme, the function `e` takes +as inputs points in `G1` and `G2` and outputs values in `Gt`. Succintly, given +some point `P` in `G1` and some point `Q` in `G1`, `e(P, Q) = C` where `C` is in `Gt`. +You can efficiently compute the mapping of points in `G1` and `G2` into `Gt`, +but you cannot efficiently determine what points were summed and paired to +produce the value in `Gt`. + +Functions are then defined to map digital signatures, messages, and keys into +and out of points of `G1` or `G2` and signature verification is the process +of calculating if a set of values representing a message, public key, and digital +signature produce the same value in `Gt` through `e`. + +Signatures can be created as either points in `G1` with public keys being +created as points in `G2` or vice versa. For the case of BLS12-381, the popular +curve used, points in `G1` are represented with 48 bytes and points in `G2` are +represented with 96 bytes. It is up to the implementer of the cryptosystem to +decide which should be larger, the public keys or the signatures. + +BLS signatures rely on pairing-based elliptic-curve cryptography to produce +various types of signatures. For a more in-depth but still high level discussion +pairing-based elliptic-curve cryptography, see Vitalik Buterin's post on +[Exploring Elliptic Curve Pairings][vitalik-pairing-post]. For much more in +depth discussion, see the specific paper on BLS12-381, [Short signatures from + the Weil Pairing][bls-weil-pairing] and +[Compact Multi-Signatures for Smaller Blockchains][multi-signatures-smaller-blockchains]. + +### Adoption + +BLS signatures have already gained traction within several popular projects. + +- Algorand is working on an implementation. +- [Zcash][zcash-adoption] has adopted BLS12-381 into the protocol. +- [Ethereum 2.0][eth-2-adoption] has adopted BLS12-381 into the protocol. +- [Chia Network][chia-adoption] has adopted BLS for signing blocks. +- [Ostracon][line-ostracon-pr], a fork of Tendermint has adopted BLS for signing blocks. + +### What systems may be affected by adding aggregated signatures? + +#### Gossip + +Gossip could be updated to aggregate vote signatures during a consensus round. +This appears to be of frankly little utility. Creating an aggregated signature +incurs overhead, so frequently re-aggregating may incur a significant +overhead. How costly this is is still subject to further investigation and +performance testing. + +Even if vote signatures were aggregated before gossip, each validator would still +need to receive and verify vote extension data from each (individual) peer validator in +order for consensus to proceed. That displaces any advantage gained by aggregating signatures across the vote message in the presence of vote extensions. + +#### Block Creation + +When creating a block, the proposer may create a small set of short +multi-signatures and attach these to the block instead of including one +signature per validator. + +#### Block Verification + +Currently, we verify each validator signature using the public key associated +with that validator. With signature aggregation, verification of blocks would +not verify many signatures individually, but would instead check the (single) +multi-signature using the public keys stored by the validator. This would also +require a mechanism for indicating which validators are included in the +aggregated signature. + +#### IBC Relaying + +IBC would no longer need to transmit a large set of signatures when +updating state. These state updates do not happen for every IBC packet, only +when changing an IBC light client's view of the counterparty chain's state. +General [IBC packets][ibc-packet] only contain enough information to correctly +route the data to the counterparty chain. + +IBC does persist commit signatures to the chain in these `MsgUpdateClient` +message when updating state. This message would no longer need the full set +of unique signatures and would instead only need one signature for all of the +data in the header. + +Adding BLS signatures would create a new signature type that must be +understood by the IBC module and by the relayers. For some operations, such +as state updates, the set of data written into the chain and received by the +IBC module could be slightly smaller. + +## Discussion + +### What are the proposed benefits to aggregated signatures? + +#### Reduce Block Size + +At the moment, a commit contains a 64-byte (512-bit) signature for each validator +that voted for the block. For the Cosmos Hub, which has 175 validators in the +active set, this amounts to about 11 KiB per block. That gives an upper bound of +around 113 GiB over the lifetime of the chain's 10.12M blocks. (Note, the Hub has +increased the number of validators in the active set over time so the total +signature size over the history of the chain is likely somewhat less than that). + +Signature aggregation would only produce two signatures for the entire block. +One for the yeas and one for the nays. Each BLS aggregated signature is 48 +bytes, per the [IETF standard of BLS signatures][bls-ietf-ecdsa-compare]. +Over the lifetime of the same Cosmos Hub chain, that would amount to about 1 +GB, a savings of 112 GB. While that is a large factor of reduction it's worth +bearing in mind that, at [GCP's cost][gcp-storage-pricing] of $.026 USD per GB, +that is a total savings of around $2.50 per month. + +#### Reduce Signature Creation and Verification Time + +From the [IETF draft standard on BLS Signatures][bls-ietf], BLS signatures can be +created in 370 microseconds and verified in 2700 microseconds. Our current +[Ed25519 implementation][voi-ed25519-perf] was benchmarked locally to take +13.9 microseconds to produce a signature and 2.03 milliseconds to batch verify +128 signatures, which is slightly fewer than the 175 in the Hub. blst, a popular +implementation of BLS signature aggregation was benchmarked to perform verification +on 100 signatures in 1.5 milliseconds [when run locally][blst-verify-bench] +on an 8 thread machine and pre-aggregated public keys. It is worth noting that +the `ed25519` library verification time grew steadily with the number of signatures, +whereas the bls library verification time remains constant. This is because the +number of operations used to verify a signature does not grow at all with the +number of signatures included in the aggregate signature (as long as the signers +signed over the same message data as is the case in Tendermint). + +It is worth noting that this would also represent a _degredation_ in signature +verification time for chains with small validator sets. When batch verifying +only 32 signatures, our ed25519 library takes .57 milliseconds, whereas BLS +would still require the same 1.5 milliseconds. + +For massive validator sets, blst dominates, taking the same 1.5 milliseconds to +check an aggregated signature from 1024 validators versus our ed25519 library's +13.066 milliseconds to batch verify a set of that size. + +#### Reduce Light-Client Verification Time + +The light client aims to be a faster and lighter-weight way to verify that a +block was voted on by a Tendermint network. The light client fetches +Tendermint block headers and commit signatures, performing public key +verification to ensure that the associated validator set signed the block. +Reducing the size of the commit signature would allow the light client to fetch +block data more quickly. + +Additionally, the faster signature verification times of BLS signatures mean +that light client verification would proceed more quickly. + +However, verification of an aggregated signature is all-or-nothing. The verifier +cannot check that some singular signer had a signature included in the block. +Instead, the verifier must use all public keys to check if some signature +was included. This does mean that any light client implementation must always +be able to fetch all public keys for any height instead of potentially being +able to check if some singular validator's key signed the block. + +#### Reduce Gossip Bandwidth + +##### Vote Gossip + +It is possible to aggregate subsets of signatures during voting, so that the +network need not gossip all _n_ validator signatures to all _n_ validators. +Theoretically, subsets of the signatures could be aggregated during consensus +and vote messages could carry those aggregated signatures. Implementing this +would certainly increase the complexity of the gossip layer but could possibly +reduce the total number of signatures required to be verified by each validator. + +##### Block Gossip + +A reduction in the block size as a result of signature aggregation would +naturally lead to a reduction in the bandwidth required to gossip a block. +Each validator would only send and receive the smaller aggregated signatures +instead of the full list of multi-signatures as we have them now. + +### What are the drawbacks to aggregated signatures? + +#### Heterogeneous key types cannot be aggregated + +Aggregation requires a specific signature algorithm, and our legacy signing schemes +cannot be aggregated. In practice, this means that aggregated signatures could +be created for a subset of validators using BLS signatures, and validators +with other key types (such as Ed25519) would still have to be be separately +propagated in blocks and votes. + +#### Many HSMs do not support aggregated signatures + +**Hardware Signing Modules** (HSM) are a popular way to manage private keys. +They provide additional security for key management and should be used when +possible for storing highly sensitive private key material. + +Below is a list of popular HSMs along with their support for BLS signatures. + +- YubiKey + - [No support][yubi-key-bls-support] +- Amazon Cloud HSM + - [No support][cloud-hsm-support] +- Ledger + - [Lists support for the BLS12-381 curve][ledger-bls-announce] + +I cannot find support listed for Google Cloud, although perhaps it exists. + +## Feasibility of implementation + +This section outlines the various hurdles that would exist to implementing BLS +signature aggregation into Tendermint. It aims to demonstrate that we _could_ +implement BLS signatures but that it would incur risk and require breaking changes for a +reasonably unclear benefit. + +### Can aggregated signatures be added as soft-upgrades? + +In my estimation, yes. With the implementation of proposer-based timestamps, +all validators now produce signatures on only one of two messages: + +1. A [CanonicalVote][canonical-vote-proto] where the BlockID is the hash of the block or +2. A `CanonicalVote` where the `BlockID` is nil. + +The block structure can be updated to perform hashing and validation in a new +way as a soft upgrade. This would look like adding a new section to the [Block.Commit][commit-proto] structure +alongside the current `Commit.Signatures` field. This new field, tentatively named +`AggregatedSignature` would contain the following structure: + +```proto +message AggregatedSignature { + // yeas is a BitArray representing which validators in the active validator + // set issued a 'yea' vote for the block. + tendermint.libs.bits.BitArray yeas = 1; + + // absent is a BitArray representing which validators in the active + // validator set did not issue votes for the block. + tendermint.libs.bits.BitArray absent = 2; + + // yea_signature is an aggregated signature produced from all of the vote + // signatures for the block. + repeated bytes yea_signature = 3; + + // nay_signature is an aggregated signature produced from all of the vote + // signatures from votes for 'nil' for this block. + // nay_signature should be made from all of the validators that were both not + // in the 'yeas' BitArray and not in the 'absent' BitArray. + repeated bytes nay_signature = 4; +} +``` + +Adding this new field as a soft upgrade would mean hashing this data structure +into the blockID along with the old `Commit.Signatures` when both are present +as well as ensuring that the voting power represented in the new +`AggregatedSignature` and `Signatures` field was enough to commit the block +during block validation. One can certainly imagine other possible schemes for +implementing this but the above should serve as a simple enough proof of concept. + +### Implementing vote-time and commit-time signature aggregation separately + +Implementing aggregated BLS signatures as part of the block structure can easily be +achieved without implementing any 'vote-time' signature aggregation. +The block proposer would gather all of the votes, complete with signatures, +as it does now, and produce a set of aggregate signatures from all of the +individual vote signatures. + +Implementing 'vote-time' signature aggregation cannot be achieved without +also implementing commit-time signature aggregation. This is because such +signatures cannot be dis-aggregated into their constituent pieces. Therefore, +in order to implement 'vote-time' signature aggregation, we would need to +either first implement 'commit-time' signature aggregation, or implement both +'vote-time' signature aggregation while also updating the block creation and +verification protocols to allow for aggregated signatures. + +### Updating IBC clients + +In order for IBC clients to function, they must be able to perform light-client +verification of blocks on counterparty chains. Because BLS signatures are not +currently part of light-clients, chains that transmit messages over IBC +cannot update to using BLS signatures without their counterparties first +being upgraded to parse and verify BLS. If chains upgrade without their +counterparties first updating, they will lose the ability to interoperate with +non-updated chains. + +### New attack surfaces + +BLS signatures and signature aggregation comes with a new set of attack surfaces. +Additionally, it's not clear that all possible major attacks are currently known +on the BLS aggregation schemes since new ones have been discovered since the ietf +draft standard was written. The known attacks are manageable and are listed below. +Our implementation would need to prevent against these but this does not appear +to present a significant hurdle to implementation. + +#### Rogue key attack prevention + +Generating an aggregated signature requires guarding against what is called +a [rogue key attack][bls-ietf-terms]. A rogue key attack is one in which a +malicious actor can craft an _aggregate_ key that can produce signatures that +appear to include a signature from a private key that the malicious actor +does not actually know. In Tendermint terms, this would look like a Validator +producing a vote signed by both itself and some other validator where the other +validator did not actually produce the vote itself. + +The main mechanisms for preventing this require that each entity prove that it +can can sign data with just their private key. The options involve either +ensuring that each entity sign a _different_ message when producing every +signature _or_ producing a [proof of possession][bls-ietf-pop] (PoP) when announcing +their key to the network. + +A PoP is a message that demonstrates ownership of a private +key. A simple scheme for PoP is one where the entity announcing +its new public key to the network includes a digital signature over the bytes +of the public key generated using the associated private key. Everyone receiving +the public key and associated proof-of-possession can easily verify the +signature and be sure the entity owns the private key. + +This PoP scheme suits the Tendermint use case quite well since +validator keys change infrequently so the associated PoPs would not be onerous +to produce, verify, and store. Using this scheme allows signature verification +to proceed more quickly, since all signatures are over identical data and +can therefore be checked using an aggregated public key instead of one at a +time, public key by public key. + +#### Summing Zero Attacks + +[Summing zero attacks][summing-zero-paper] are attacks that rely on using the '0' point of an +elliptic curve. For BLS signatures, if the point 0 is chosen as the private +key, then the 0 point will also always be the public key and all signatures +produced by the key will also be the 0 point. This is easy enough to +detect when verifying each signature individually. + +However, because BLS signature aggregation creates an aggregated signature and +an aggregated public key, a set of colluding signers can create a pair or set +of signatures that are non-zero but which aggregate ("sum") to 0. The signatures that sum zero along with the +summed public key of the colluding signers will verify any message. This would +allow the colluding signers to sign any block or message with the same signature. +This would be reasonably easy to detect and create evidence for because, in +all other cases, the same signature should not verify more than message. It's +not exactly clear how such an attack would advantage the colluding validators +because the normal mechanisms of evidence gathering would still detect the +double signing, regardless of the signatures on both blocks being identical. + +### Backwards Compatibility + +Backwards compatibility is an important consideration for signature verification. +Specifically, it is important to consider whether chains using current versions +of IBC would be able to interact with chains adopting BLS. + +Because the `Block` shared by IBC and Tendermint is produced and parsed using +protobuf, new structures can be added to the Block without breaking the +ability of legacy users to parse the new structure. Breaking changes between +current users of IBC and new Tendermint blocks only occur if data that is +relied upon by the current users is no longer included in the current fields. + +For the case of BLS aggregated signatures, a new `AggregatedSignature` field +can therefore be added to the `Commit` field without breaking current users. +Current users will be broken when counterparty chains upgrade to the new version +and _begin using_ BLS signatures. Once counterparty chains begin using BLS +signatures, the BlockID hashes will include hashes of the `AggregatedSignature` +data structure that the legacy users will not be able to compute. Additionally, +the legacy software will not be able to parse and verify the signatures to +ensure that a supermajority of validators from the counterparty chain signed +the block. + +### Library Support + +Libraries for BLS signature creation are limited in number, although active +development appears to be ongoing. Cryptographic algorithms are difficult to +implement correctly and correctness issues are extremely serious and dangerous. +No further exploration of BLS should be undertaken without strong assurance of +a well-tested library with continuing support for creating and verifying BLS +signatures. + +At the moment, there is one candidate, `blst`, that appears to be the most +mature and well vetted. While this library is undergoing continuing auditing +and is supported by funds from the Ethereum foundation, adopting a new cryptographic +library presents some serious risks. Namely, if the support for the library were +to be discontinued, Tendermint may become saddled with the requirement of supporting +a very complex piece of software or force a massive ecosystem-wide migration away +from BLS signatures. + +This is one of the more serious reasons to avoid adopting BLS signatures at this +time. There is no gold standard library. Some projects look promising, but no +project has been formally verified with a long term promise of being supported +well into the future. + +#### Go Standard Library + +The Go Standard library has no implementation of BLS signatures. + +#### BLST + +[blst][blst], or 'blast' is an implementation of BLS signatures written in C +that provides bindings into Go as part of the repository. This library is +actively undergoing formal verification by Galois and previously received an +initial audit by NCC group, a firm I'd never heard of. + +`blst` is [targeted for use in prysm][prysm-blst], the golang implementation of Ethereum 2.0. + +#### Gnark-Crypto + +[Gnark-Crypto][gnark] is a Go-native implementation of elliptic-curve pairing-based +cryptography. It is not audited and is documented as 'as-is', although +development appears to be active so formal verification may be forthcoming. + +#### CIRCL + +[CIRCL][circl] is a go-native implementation of several cryptographic primitives, +bls12-381 among them. The library is written and maintained by Cloudflare and +appears to receive frequent contributions. However, it lists itself as experimental +and urges users to take caution before using it in production. + +### Added complexity to light client verification + +Implementing BLS signature aggregation in Tendermint would pose issues for the +light client. The light client currently validates a subset of the signatures +on a block when performing the verification algorithm. This is no longer possible +with an aggregated signature. Aggregated signature verification is all-or-nothing. +The light client could no longer check that a subset of validators from some +set of validators is represented in the signature. Instead, it would need to create +a new aggregated key with all the stated signers for each height it verified where +the validator set changed. + +This means that the speed advantages gained by using BLS cannot be fully realized +by the light client since the client needs to perform the expensive operation +of re-aggregating the public key. Aggregation is _not_ constant time in the +number of keys and instead grows linearly. When [benchmarked locally][blst-verify-bench-agg], +blst public key aggregation of 128 keys took 2.43 milliseconds. This, along with +the 1.5 milliseconds to verify a signature would raise light client signature +verification time to 3.9 milliseconds, a time above the previously mentioned +batch verification time using our ed25519 library of 2.0 milliseconds. + +Schemes to cache aggregated subsets of keys could certainly cut this time down at the +cost of adding complexity to the light client. + +### Added complexity to evidence handling + +Implementing BLS signature aggregation in Tendermint would add complexity to +the evidence handling within Tendermint. Currently, the light client can submit +evidence of a fork attempt to the chain. This evidence consists of the set of +validators that double-signed, including their public keys, with the conflicting +block. + +We can quickly check that the listed validators double signed by verifying +that each of their signatures are in the submitted conflicting block. A BLS +signature scheme would change this by requiring the light client to submit +the public keys of all of the validators that signed the conflicting block so +that the aggregated signature may be checked against the full signature set. +Again, aggregated signature verification is all-or-nothing, so without all of +the public keys, we cannot verify the signature at all. These keys would be +retrievable. Any party that wanted to create a fork would want to convince a +network that its fork is legitimate, so it would need to gossip the public keys. +This does not hamper the feasibility of implementing BLS signature aggregation +into Tendermint, but does represent yet another piece of added complexity to +the associated protocols. + +## Open Questions + +- _Q_: Can you aggregate Ed25519 signatures in Tendermint? + - There is a suggested scheme in github issue [7892][suggested-ed25519-agg], +but additional rigor would be required to fully verify its correctness. + +## Current Consideration + +Adopting a signature aggregation scheme presents some serious risks and costs +to the Tendermint project. It requires multiple backwards-incompatible changes +to the code, namely a change in the structure of the block and a new backwards-incompatible +signature and key type. It risks adding a new signature type for which new attack +types are still being discovered _and_ for which no industry standard, battle-tested +library yet exists. + +The gains boasted by this new signing scheme are modest: Verification time is +marginally faster and block sizes shrink by a few kilobytes. These are relatively +minor gains in exchange for the complexity of the change and the listed risks of the technology. +We should take a wait-and-see approach to BLS signature aggregation, monitoring +the up-and-coming projects and consider implementing it as the libraries and +standards develop. + +### References + +[line-ostracon-pr]: https://github.com/line/ostracon/pull/117 +[gcp-storage-pricing]: https://cloud.google.com/storage/pricing#north-america_2 +[yubi-key-bls-support]: https://github.com/Yubico/yubihsm-shell/issues/66 +[cloud-hsm-support]: https://docs.aws.amazon.com/cloudhsm/latest/userguide/pkcs11-key-types.html +[bls-ietf]: https://datatracker.ietf.org/doc/html/draft-irtf-cfrg-bls-signature-04 +[bls-ietf-terms]: https://datatracker.ietf.org/doc/html/draft-irtf-cfrg-bls-signature-04#section-1.3 +[bls-ietf-pop]: https://datatracker.ietf.org/doc/html/draft-irtf-cfrg-bls-signature-04#section-3.3 +[multi-signatures-smaller-blockchains]: https://eprint.iacr.org/2018/483.pdf +[zcash-adoption]: https://github.com/zcash/zcash/issues/2502 +[chia-adoption]: https://github.com/Chia-Network/chia-blockchain#chia-blockchain +[bls-ietf-ecdsa-compare]: https://datatracker.ietf.org/doc/html/draft-irtf-cfrg-bls-signature-04#section-1.1 +[voi-ed25519-perf]: https://github.com/williambanfield/curve25519-voi/blob/benchmark/primitives/ed25519/PERFORMANCE.txt#L79 +[blst-verify-bench]: https://github.com/williambanfield/blst/blame/bench/bindings/go/PERFORMANCE.md#L9 +[blst-verify-bench-agg]: https://github.com/williambanfield/blst/blame/bench/bindings/go/PERFORMANCE.md#L23 +[vitalik-pairing-post]: https://medium.com/@VitalikButerin/exploring-elliptic-curve-pairings-c73c1864e627 +[ledger-bls-announce]: https://www.ledger.com/first-ever-firmware-update-coming-to-the-ledger-nano-x +[commit-proto]: https://github.com/tendermint/tendermint/blob/be7cb50bb3432ee652f88a443e8ee7b8ef7122bc/proto/tendermint/types/types.proto#L121 +[canonical-vote-proto]: https://github.com/tendermint/tendermint/blob/be7cb50bb3432ee652f88a443e8ee7b8ef7122bc/spec/core/encoding.md#L283 +[blst]: https://github.com/supranational/blst +[prysm-blst]: https://github.com/prysmaticlabs/prysm/blob/develop/go.mod#L75 +[gnark]: https://github.com/ConsenSys/gnark-crypto/ +[eth-2-adoption]: https://notes.ethereum.org/@GW1ZUbNKR5iRjjKYx6_dJQ/Skxf3tNcg_ +[bls-weil-pairing]: https://www.iacr.org/archive/asiacrypt2001/22480516.pdf +[summing-zero-paper]: https://eprint.iacr.org/2021/323.pdf +[circl]: https://github.com/cloudflare/circl +[suggested-ed25519-agg]: https://github.com/tendermint/tendermint/issues/7892 diff --git a/cometbft/v0.38/docs/rfc/tendermint-core/rfc-019-config-version.md b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-019-config-version.md new file mode 100644 index 00000000..5bf0c104 --- /dev/null +++ b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-019-config-version.md @@ -0,0 +1,401 @@ + +# RFC 019: Configuration File Versioning + +## Changelog + +- 19-Apr-2022: Initial draft (@creachadair) +- 20-Apr-2022: Updates from review feedback (@creachadair) + +## Abstract + +Updating configuration settings is an essential part of upgrading an existing +node to a new version of the Tendermint software. Unfortunately, it is also +currently a very manual process. This document discusses some of the history of +changes to the config format, actions we've taken to improve the tooling for +configuration upgrades, and additional steps we may want to consider. + +## Background + +A Tendermint node reads configuration settings at startup from a TOML formatted +text file, typically named `config.toml`. The contents of this file are defined +by the [`github.com/tendermint/tendermint/config`][config-pkg]. + +Although many settings in this file remain valid from one version of Tendermint +to the next, new versions of Tendermint often add, update, and remove settings. +These changes often require manual intervention by operators who are upgrading +their nodes. + +I propose we should provide better tools and documentation to help operators +make configuration changes correctly during version upgrades. Ideally, as much +as possible of any configuration file update should be automated, and where +that is not possible or practical, we should provide clear, explicit directions +for what steps need to be taken manually. Moreover, when the node discovers +incorrect or invalid configuration, we should improve the diagnostics it emits +so that the operator can quickly and easily find the relevant documentation, +without having to grep through source code. + +## Discussion + +By convention, we are supposed to document required changes to the config file +in the `UPGRADING.md` file for the release that introduces them. Although we +have mostly done this, the level of detail in the upgrading instructions is +often insufficient for an operator to correctly update their file. + +The updates vary widely in complexity: Operators may need to add new required +settings, update obsolete values for existing settings, move or rename existing +settings within the file, or remove obsolete settings (which are thus invalid). +Here are a few examples of each of these cases: + +- **New required settings:** Tendermint v0.35 added a new top-level `mode` + setting that determines whether a node runs as a validator, a full node, or a + seed node. The default value is `"full"`, which means the operator of a + validator must manually add `mode = "validator"` (or set the `--mode` flag on + the command line) for their node to come up in the correct mode. + +- **Updated obsolete values:** Tendermint v0.35 removed support for versions + `"v1"` and `"v2"` of the blocksync (formerly "fastsync") protocol, requiring + any node using either of those values to update to `"v0"`. + +- **Moved/renamed settings:** Version v0.34 moved the top-level `pprof_laddr` + setting under the `[rpc]` section. + + Version v0.35 renamed every setting in the file from `snake_case` to + `kebab-case`, moved the top-level `fast_sync` setting into the `[blocksync]` + section as (itself renamed from `[fastsync]`), and moved all the top-level + `priv-validator-*` settings under a new `[priv-validator]` section with their + prefix trimmed off. + +- **Removed obsolete settings:** Version v0.34 removed the `index_all_keys` and + `index_keys` settings from the `[tx_index]` section; version v0.35 removed + the `wal-dir` setting from the `[mempool]` section, and version v0.36 removed + the `[blocksync]` section entirely. + +While many of these changes are mentioned in the config section of the upgrade +instructions, some are not mentioned at all, or are hidden in other parts of +the doc. For instance, the v0.34 `pprof_laddr` change was documented only as an +RPC flag change. (A savvy reader might realize that the flag `--rpc.pprof_laddr` +implies a corresponding config section, but it omits the related detail that +there was a top-level setting that's been renamed). The lesson here is not +that the docs are bad, but to point out that prose is not the most efficient +format to convey detailed changes like this. The upgrading instructions are +still valuable for the human reader to understand what to expect. + +### Concrete Steps + +As part of the v0.36 development cycle, we spent some time reverse-engineering +the configuration changes since the v0.34 release and built an experimental +command-line tool called [`confix`][confix], whose job it is to automatically +update the settings in a `config.toml` file to the latest version. We also +backported a version of this tool into the v0.35.x branch at release v0.35.4. + +This tool should work fine for configuration files created by Tendermint v0.34 +and later, but does not (yet) know how to handle changes from prior versions of +Tendermint. Part of the difficulty for older versions is simply logistical: To +figure out which changes to apply, we need to understand something about the +version that made the file, as well as the version we're converting it to. + +> **Discussion point:** In the future we might want to consider incorporating +> this into the node CLI directly, but we're keeping it separate for now until +> we can get some feedback from operators. + +For the experiment, we handled this by carefully searching the history of +config format changes for shibboleths to bound the version: For example, the +`[fastsync]` section was added in Tendermint v0.32 and renamed `[blocksync]` in +Tendermint v0.35. So if we see a `[fastsync]` section, we have some confidence +that the file was created by v0.32, v0.33, or v0.34. + +But such signals are delicate: The `[blocksync]` section was removed in v0.36, +so if we do not find `[fastsync]`, we cannot conclude from that alone that the +file is from v0.31 or earlier -- we have to look for corroborating details. +While such "sniffing" tactics are fine for an experiment, they aren't as robust +as we might like. + +This is especially relevant for configuration files that may have already been +manually upgraded across several versions by the time we are asked to update +them again. Another related concern is that we'd like to make sure conversion +is idempotent, so that it would be safe to rerun the tool over an +already-converted file without breaking anything. + +### Config Versioning + +One obvious tactic we could use for future releases is add a version marker to +the config file. This would give tools like `confix` (and the node itself) a +way to calibrate their expectations. Rather than being a version for the file +itself, however, this version marker would indicate which version of Tendermint +is needed to read the file. + +Provisionally, this might look something like: + +```toml +# THe minimum version of Tendermint compatible with the contents of +# this configuration file. +config-version = 'v0.35' +``` + +When initializing a new node, Tendermint would populate this field with its own +version (e.g., `v0.36`). When conducting an upgrade, tools like `confix` can +then use this to decide which conversions are valid, and then update the value +accordingly. After converting a file marked `'v0.35'` to`'v0.37'`, the +conversion tool sets the file's `config-version` to reflect its compatibility. + +> **Discussion point:** This example presumes we would keep config files +> compatible within a given release cycle, e.g., all of v0.36.x. We could also +> use patch numbers here, if we think there's some reason to permit changes +> that would require config file edits at that granularity. I don't think we +> should, but that's a design question to consider. + +Upon seeing an up-to-date version marker, the conversion tool can simply exit +with a diagnostic like "this file is already up-to-date", rather than sniffing +the keyspace and potentially introducing errors. In addition, this would let a +tool detect config files that are _newer_ than the one it understands, and +issue a safe diagnostic rather than doing something wrong. Plus, besides +avoiding potentially unsafe conversions, this would also serve as +human-readable documentation that the file is up-to-date for a given version. + +Adding a config version would not address the problem of how to convert files +created by older versions of Tendermint, but it would at least help us build +more robust config tooling going forward. + +### Stability and Change + +In light of the discussion so far, it is natural to examine why we make so many +changes to the configuration file from one version to the next, and whether we +could reduce friction by being more conservative about what we make +configurable, what config changes we make over time, and how we roll them out. + +Some changes, like renaming everything from snake case to kebab case, are +entirely gratuitous. We could safely agree not to make those kinds of changes. +Apart from that obvious case, however, many other configuration settings +provide value to node operators in cases where there is no simple, universal +setting that matches every application. + +Taking a high-level view, there are several broad reasons why we might want to +make changes to configuration settings: + +- **Lessons learned:** Configuration settings are a good way to try things out + in production, before making more invasive changes to the consensus protocol. + + For example, up until Tendermint v0.35, consensus timeouts were specified as + per-node configuration settings (e.g., `timeout-precommit` et al.). This + allowed operators to tune these values for the needs of their network, but + had the downside that individually-misconfigured nodes could stall consensus. + + Based on that experience, these timeouts have been deprecated in Tendermint + v0.36 and converted to consensus parameters, to be consistent across all + nodes in the network. + +- **Migration & experimentation:** Introducing new features and updating old + features can complicate migration for existing users of the software. + Temporary or "experimental" configuration settings can be a valuable way to + mitigate that friction. + + For example, Tendermint v0.36 introduces a new RPC event subscription + endpoint (see [ADR 075][adr075]) that will eventually replace the existing + webwocket-based interface. To give users time to migrate, v0.36 adds an + `experimental-disable-websocket` setting, defaulted to `false`, that allows + operators to selectively disable the websocket API for testing purposes + during the conversion. This setting is designed to be removed in v0.37, when + the old interface is no longer supported. + +- **Ongoing maintenance:** Sometimes configuration settings become obsolete, + and the cost of removing them trades off against the potential risks of + leaving a non-functional or deprecated knob hooked up indefinitely. + + For example, Tendermint v0.35 deprecated two alternate implementations of the + blocksync protocol, one of which was deleted entirely (`v1`) and one of which + was scheduled for removal (`v2`). The `blocksync.version` setting, which had + been added as a migration aid, became obsolete and needed to be updated. + + Despite our best intentions, sometimes engineering designs do not work out. + It's just as important to leave room to back out of changes we have since + reconsidered, as it is to support migrations forward onto new and improved + code. + +- **Clarity and legibility:** Besides configuring the software, another + important purpose of a config file is to document intent for the humans who + operate and maintain the software. Operators need adjust settings to keep the + node running, and developers need to know what options were in use when + something goes wrong so they can diagnose and fix bugs. The legibility of a + config file as a _human_ artifact is also thus important. + + For example, Tendermint v0.35 moved settings related to validator private + keys from the top-level section of the configuration file to their own + designated `[priv-validator]` section. Although this change did not make any + difference to the meaning of those settings, it made the organization of the + file easier to understand, and allowed the names of the individual settings + to be simplified (e.g., `priv-validator-key-file` became simply `key-file` in + the new section). + + Although such changes are "gratuitous" with respect to the software, there is + often value in making things more legible for the humans. While there is no + simple rule to define the line, the Potter Stewart principle can be used with + due care. + +Keeping these examples in mind, we can and should take reasonable steps to +avoid churn in the configuration file across versions where we can. However, we +must also accept that part of the reason for _having_ a config file is to allow +us flexibility elsewhere in the design. On that basis, we should not attempt +to be too dogmatic about config changes either. Unlike changes in the block +protocol, for example, which affect every user of every network that adopts +them, config changes are relatively self-contained. + +There are few guiding principles I think we can use to strike a sensible +balance: + +1. **No gratuitous changes.** Aesthetic changes that do not enhance legibility, + avert confusion, or clarity documentation, should be entirely avoided. + +2. **Prefer mechanical changes.** Whenever it is practical, change settings in + a way that can be updated by a tool without operator judgement. This implies + finding safe, universal defaults for new settings, and not changing the + default values of existing settings. + + Even if that means we have to make multiple changes (e.g., add a new setting + in the current version, deprecate the old one, and remove the old one in the + next version) it's preferable if we can mechanize each step. + +3. **Clearly signal intent.** When adding temporary or experimental settings, + they should be clearly named and documented as such. Use long names and + suggestive prefixes (e.g., `experimental-*`) so that they stand out when + read in the config file or printed in logs. + + Relatedly, using temporary or experimental settings should cause the + software to emit diagnostic logs at runtime. These log messages should be + easy to grep for, and should contain pointers to more complete documentation + (say, issue numbers or URLs) that the operator can read, as well as a hint + about when the setting is expected to become invalid. For example: + + ``` + WARNING: Websocket RPC access is deprecated and will be removed in + Tendermint v0.37. See https://tinyurl.com/adr075 for more information. + ``` + +4. **Consider both directions.** When adding a configuration setting, take some + time during the implementation process to think about how the setting could + be removed, as well as how it will be rolled out. This applies even for + settings we imagine should be permanent. Experience may cause is to rethink + our original design intent more broadly than we expected. + + This does not mean we have to spend a long time picking nits over the design + of every setting; merely that we should convince ourselves we _could_ undo + it without making too big a mess later. Even a little extra effort up front + can sometimes save a lot. + +## References + +- [Tendermint `config` package][config-pkg] +- [`confix` command-line tool][confix] +- [`condiff` command-line tool][condiff] +- [Configuration update plan][plan] +- [ADR 075: RPC Event Subscription Interface][adr075] + +[config-pkg]: https://godoc.org/github.com/tendermint/tendermint/config +[confix]: https://github.com/tendermint/tendermint/blob/main/scripts/confix +[condiff]: https://github.com/tendermint/tendermint/blob/main/scripts/confix/condiff +[plan]: https://github.com/tendermint/tendermint/blob/main/scripts/confix/plan.go +[testdata]: https://github.com/tendermint/tendermint/blob/main/scripts/confix/testdata +[adr075]: https://github.com/tendermint/tendermint/blob/main/docs/architecture/adr-075-rpc-subscription.md + +## Appendix: Research Notes + +Discovering when various configuration settings were added, updated, and +removed turns out to be surprisingly tedious. To solve this puzzle, we had to +answer the following questions: + +1. What changes were made between v0.x and v0.y? This is further complicated by + cases where we have backported config changes into the middle of an earlier + release cycle (e.g., `psql-conn` from v0.35.x into v0.34.13). + +2. When during the development cycle were those changes made? This allows us to + recognize features that were backported into a previous release. + +3. What were the default values of the changed settings, and did they change at + all during or across the release boundary? + +Each step of the [configuration update plan][plan] is commented with a link to +one or more PRs where that change was made. The sections below discuss how we +found these references. + +### Tracking Changes Across Releases + +To figure out what changed between two releases, we built a tool called +[`condiff`][condiff], which performs a "keyspace" diff of two TOML documents. +This diff respects the structure of the TOML file, but ignores comments, blank +lines, and configuration values, so that we can see what was added and removed. + +To use it, run: + +```shell +go run ./scripts/confix/condiff old.toml new.toml +``` + +This tool works on any TOML documents, but for our purposes we needed +Tendermint `config.toml` files. The easiest way to get these is to build the +node binary for your version of interest, run `tendermint init` on a clean home +directory, and copy the generated config file out. The [`testdata`][testdata] +directory for the `confix` tool has configs generated from the heads of each +release branch from v0.31 through v0.35. + +If you want to reproduce this yourself, it looks something like this: + +```shell +# Example for Tendermint v0.32. +git checkout --track origin/v0.32.x +go get golang.org/x/sys/unix +go mod tidy +make build +rm -fr -- tmhome +./build/tendermint --home=tmhome init +cp tmhome/config/config.toml config-v32.toml +``` + +Be advised that the further back you go, the more idiosyncrasies you will +encounter. For example, Tendermint v0.31 and earlier predate Go modules (v0.31 +used dep), and lack backport branches. And you may need to do some editing of +Makefile rules once you get back into the 20s. + +Note that when diffing config files across the v0.34/v0.35 gap, the swap from +`snake_case` to `kebab-case` makes it look like everything changed. The +`condiff` tool has a `-desnake` flag that normalizes all the keys to kebab case +in both inputs before comparison. + +### Locating Additions and Deletions + +To figure out when a configuration setting was added or removed, your tool of +choice is `git bisect`. The only tricky part is finding the endpoints for the +search. If the transition happened within a release, you can use that +release's backport branch as the endpoint (if it has one, e.g., `v0.35.x`). + +However, the start point can be more problematic. The backport branches are not +ancestors of `master` or of each other, which means you need to find some point +in history _prior_ to the change but still attached to the mainline. For recent +releases there is a dev root (e.g., `v0.35.0-dev`, `v0.34.0-dev1`, etc.). These +are not named consistently, but you can usually grep the output of `git tag` to +find them. + +In the worst case you could try starting from the root commit of the repo, but +that turns out not to work in all cases. We've done some branching shenanigans +over the years that mean the root is not a direct ancestor of all our release +branches. When you find this you will probably swear a lot. I did. + +Once you have a start and end point (say, `v0.35.0-dev` and `master`), you can +bisect in the usual way. I use `git grep` on the `config` directory to check +whether the case I am looking for is present. For example, to find when the +`[fastsync]` section was removed: + +```shell +# Setup: +git checkout master +git bisect start +git bisect bad # it's not present on tip of master. +git bisect good v0.34.0-dev1 # it was present at the start of v0.34. +``` + +```shell +# Now repeat this until it gives you a specific commit: +if git grep -q '\[fastsync\]' config ; then git bisect good ; else git bisect bad ; fi +``` + +The above example finds where a config was removed: To find where a setting was +added, do the same thing except reverse the sense of the test (`if ! git grep -q +...`). diff --git a/cometbft/v0.38/docs/rfc/tendermint-core/rfc-020-onboarding-projects.rst b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-020-onboarding-projects.rst new file mode 100644 index 00000000..d9f883ca --- /dev/null +++ b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-020-onboarding-projects.rst @@ -0,0 +1,240 @@ +======================================= +RFC 020: Tendermint Onboarding Projects +======================================= + +.. contents:: + :backlinks: none + +Changelog +--------- + +- 2022-03-30: Initial draft. (@tychoish) +- 2022-04-25: Imported document to tendermint repository. (@tychoish) + +Overview +-------- + +This document describes a collection of projects that might be good for new +engineers joining the Tendermint Core team. These projects mostly describe +features that we'd be very excited to see land in the code base, but that are +intentionally outside of the critical path of a release on the roadmap, and +have the following properties that we think make good on-boarding projects: + +- require relatively little context for the project or its history beyond a + more isolated area of the code. + +- provide exposure to different areas of the codebase, so new team members + will have reason to explore the code base, build relationships with people + on the team, and gain experience with more than one area of the system. + +- be of moderate size, striking a healthy balance between trivial or + mechanical changes (which provide little insight) and large intractable + changes that require deeper insight than is available during onboarding to + address well. A good size project should have natural touchpoints or + check-ins. + +Projects +-------- + +Before diving into one of these projects, have a conversation about the +project or aspects of Tendermint that you're excited to work on with your +onboarding buddy. This will help make sure that these issues are still +relevant, help you get any context, underatnding known pitfalls, and to +confirm a high level approach or design (if relevant.) On-boarding buddies +should be prepared to do some design work before someone joins the team. + +The descriptions that follow provide some basic background and attempt to +describe the user stories and the potential impact of these project. + +E2E Test Systems +~~~~~~~~~~~~~~~~ + +Tendermint's E2E framework makes it possible to run small test networks with +different Tendermint configurations, and make sure that the system works. The +tests run Tendermint in a separate binary, and the system provides some very +high level protection against making changes that could break Tendermint in +otherwise difficult to detect ways. + +Working on the E2E system is a good place to get introduced to the Tendermint +codebase, particularly for developers who are newer to Go, as the E2E +system (generator, runner, etc.) is distinct from the rest of Tendermint and +comparatively quite small, so it may be easier to begin making changes in this +area. At the same time, because the E2E system exercises *all* of Tendermint, +work in this area is a good way to get introduced to various components of the +system. + +Configurable E2E Workloads +++++++++++++++++++++++++++ + +All E2E tests use the same workload (e.g. generated transactions, submitted to +different nodes in the network,) which has been tuned empirically to provide a +gentle but consistent parallel load that all E2E tests can pass. Ideally, the +workload generator could be configurable to have different shapes of work +(bursty, different transaction sizes, weighted to different nodes, etc.) and +even perhaps further parameterized within a basic shape, which would make it +possible to use our existing test infrastructure to answer different questions +about the performance or capability of the system. + +The work would involve adding a new parameter to the E2E test manifest, and +creating an option (e.g. "legacy") for the current load generation model, +extract configurations options for the current load generation, and then +prototype implementations of alternate load generation, and also run some +preliminary using the tools. + +Byzantine E2E Workloads ++++++++++++++++++++++++ + +There are two main kinds of integration tests in Tendermint: the E2E test +framework, and then a collection of integration tests that masquerade as +unit-tests. While some of this expansion of test scope is (potentially) +inevitable, the masquerading unit tests (e.g ``consensus.byzantine_test.go``) +end up being difficult to understand, difficult to maintain, and unreliable. + +One solution to this, would be to modify the E2E ABCI application to allow it +to inject byzantine behavior, and then have this be a configurable aspect of +a test network to be able to provoke Byzantine behavior in a "real" system and +then observe that evidence is constructed. This would make it possible to +remove the legacy tests entirely once the new tests have proven themselves. + +Abstract Orchestration Framework +++++++++++++++++++++++++++++++++ + +The orchestration of e2e test processes is presently done using docker +compose, which works well, but has proven a bit limiting as all processes need +to run on a single machine, and the log aggregation functions are confusing at +best. + +This project would replace the current orchestration with something more +generic, potentially maintaining the current system, but also allowing the e2e +tests to manage processes using k8s. There are a few "local" k8s frameworks +(e.g. kind and k3s,) which might be able to be useful for our current testing +model, but hopefully, we could use this new implementation with other k8s +systems for more flexible distribute test orchestration. + +Improve Operationalize Experience of ``run-multiple.sh`` +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + +The e2e test runner currently runs a single test, and in most cases we manage +the test cases using a shell script that ensure cleanup of entire test +suites. This is a bit difficult to maintain and makes reproduction of test +cases more awkward than it should be. The e2e ``runner`` itself should provide +equivalent functionality to ``run-multiple.sh``: ensure cleanup of test cases, +collect and process output, and be able to manage entire suites of cases. + +It might also be useful to implement an e2e test orchestrator that runs all +tendermint instances in a single process, using "real" networks for faster +feedback and iteration during development. + +In addition to being a bit easier to maintain, having a more capable runner +implementation would make it easier to collect data from test runs, improve +debugability and reporting. + +Fan-Out For CI E2E Tests +++++++++++++++++++++++++ + +While there are some parallelism in the execution of e2e tests, each e2e test +job must build a tendermint e2e image, which takes about 5 minutes of CPU time +per-task, which given the size of each of the runs. + +We'd like to be able to reduce the amount of overhead per-e2e tests while +keeping the cycle time for working with the tests very low, while also +maintaining a reasonable level of test coverage. This is an impossible +tradeoff, in some ways, and the percentage of overhead at the moment is large +enough that we can make some material progress with a moderate amount of time. + +Most of this work has to do with modifying github actions configuration and +e2e artifact (docker) building to reduce redundant work. Eventually, when we +can drop the requirement for CGo storage engines, it will be possible to move +(cross) compile tendermint locally, and then inject the binary into the docker +container, which would reduce a lot of the build-time complexity, although we +can move more in this direction or have runtime flags to disable CGo +dependencies for local development. + +Remove Panics +~~~~~~~~~~~~~ + +There are lots of places in the code base which can panic, and would not be +particularly well handled. While in some cases, panics are the right answer, +in many cases the panics were just added to simplify downstream error +checking, and could easily be converted to errors. + +The `Don't Panic RFC +<./rfc-008-do-not-panic.md>`_ +covers some of the background and approach. + +While the changes are in this project are relatively rote, this will provide +exposure to lots of different areas of the codebase as well as insight into +how different areas of the codebase interact with eachother, as well as +experience with the test suites and infrastructure. + +Implement more Expressive ABCI Applications +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Tendermint maintains two very simple ABCI applications (a KV application used +for basic testing, and slightly more advanced test application used in the +end-to-end tests). Writing an application would provide a new engineer with +useful experiences using Tendermint that mirrors the expierence of downstream +users. + +This is more of an exploratory project, but could include providing common +interfaces on top of Tendermint consensus for other well known protocols or +tools (e.g. ``etcd``) or a DNS server or some other tool. + +Self-Regulating Reactors +~~~~~~~~~~~~~~~~~~~~~~~~ + +Currently reactors (the internal processes that are responsible for the higher +level behavior of Tendermint) can be started and stopped, but have no +provision for being paused. These additional semantics may allow Tendermint to +pause reactors (and avoid processing their messhages, etc.) and allow better +coordination in the future. + +While this is a big project, it's possible to break this apart into many +smaller projects: make p2p channels pauseable, add pause/UN-pause hooks to the +service implementation and machinery, and finally to modify the reactor +implementations to take advantage of these additional semantics + +This project would give an engineer some exposure to the p2p layer of the +code, as well as to various aspects of the reactor implementations. + +Metrics +~~~~~~~ + +Tendermint has a metrics system that is relatively underutilized, and figuring +out ways to capture and organize the metrics to provide value to users might +provide an interesting set of projects for new engineers on Tendermint. + +Convert Logs to Metrics ++++++++++++++++++++++++ + +Because the tendermint logs tend to be quite verbose and not particularly +actionable, most users largely ignore the logging or run at very low +verbosity. While the log statements in the code do describe useful events, +taken as a whole the system is not particularly tractable, and particularly at +the Debug level, not useful. One solution to this problem is to identify log +messages that might be (e.g. increment a counter for certian kinds of errors) + +One approach might be to look at various logging statements, particularly +debug statements or errors that are logged but not returned, and see if +they're convertable to counters or other metrics. + +Expose Metrics to Tests ++++++++++++++++++++++++ + +The existing Tendermint test suites replace the metrics infrastructure with +no-op implementations, which means that tests can neither verify that metrics +are ever recorded, nor can tests use metrics to observe events in the +system. Writing an implementation, for testing, that makes it possible to +record metrics and provides an API for introspecting this data, as well as +potentially writing tests that take advantage of this type, could be useful. + +Logging Metrics ++++++++++++++++ + +In some systems, the logging system itself can provide some interesting +insights for operators: having metrics that track the number of messages at +different levels as well as the total number of messages, can act as a canary +for the system as a whole. + +This should be achievable by adding an interceptor layer within the logging +package itself that can add metrics to the existing system. diff --git a/cometbft/v0.38/docs/rfc/tendermint-core/rfc-021-socket-protocol.md b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-021-socket-protocol.md new file mode 100644 index 00000000..4b8fd2ab --- /dev/null +++ b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-021-socket-protocol.md @@ -0,0 +1,266 @@ +# RFC 021: The Future of the Socket Protocol + +## Changelog + +- 19-May-2022: Initial draft (@creachadair) +- 19-Jul-2022: Converted from ADR to RFC (@creachadair) + +## Abstract + +This RFC captures some technical discussion about the ABCI socket protocol that +was originally documented to solicit an architectural decision. This topic was +not high-enough priority as of this writing to justify making a final decision. + +For that reason, the text of this RFC has the general structure of an ADR, but +should be viewed primarily as a record of the issue for future reference. + +## Background + +The [Application Blockchain Interface (ABCI)][abci] is a client-server protocol +used by the Tendermint consensus engine to communicate with the application on +whose behalf it performs state replication. There are currently three transport +options available for ABCI applications: + +1. **In-process**: Applications written in Go can be linked directly into the + same binary as the consensus node. Such applications use a "local" ABCI + connection, which exposes application methods to the node as direct function + calls. + +2. **Socket protocol**: Out-of-process applications may export the ABCI service + via a custom socket protocol that sends requests and responses over a + Unix-domain or TCP socket connection as length-prefixed protocol buffers. + In Tendermint, this is handled by the [socket client][socket-client]. + +3. **gRPC**: Out-of-process applications may export the ABCI service via gRPC. + In Tendermint, this is handled by the [gRPC client][grpc-client]. + +Both the out-of-process options (2) and (3) have a long history in Tendermint. +The beginnings of the gRPC client were added in [May 2016][abci-start] when +ABCI was still hosted in a separate repository, and the socket client (formerly +called the "remote client") was part of ABCI from its inception in November +2015. + +At that time when ABCI was first being developed, the gRPC project was very new +(it launched Q4 2015) and it was not an obvious choice for use in Tendermint. +It took a while before the language coverage and quality of gRPC reached a +point where it could be a viable solution for out-of-process applications. For +that reason, it made sense for the initial design of ABCI to focus on a custom +protocol for out-of-process applications. + +## Problem Statement + +For practical reasons, ABCI needs an interprocess communication option to +support applications not written in Go. The two practical options are RPC and +FFI, and for operational reasons an RPC mechanism makes more sense. + +The socket protocol has not changed all that substantially since its original +design, and has the advantage of being simple to implement in almost any +reasonable language. However, its simplicity includes some limitations that +have had a negative impact on the stability and performance of out-of-process +applications using it. In particular: + +- The protocol lacks request identifiers, so the client and server must return + responses in strict FIFO order. Even if the client issues requests that have + no dependency on each other, the protocol has no way except order of issue to + map responses to requests. + + This reduces (in some cases substantially) the concurrency an application can + exploit, since the parallelism of requests in flight is gated by the slowest + active request at any moment. There have been complaints from some network + operators on that basis. + +- The protocol lacks method identifiers, so the only way for the client and + server to understand which operation is requested is to dispatch on the type + of the request and response payloads. For responses, this means that [any + error condition is terminal not only to the request, but to the entire ABCI + client](https://github.com/tendermint/tendermint/blob/main/abci/client/socket_client.go#L149). + + The historical intent of terminating for any error seems to have been that + all ABCI errors are unrecoverable and hence protocol fatal + (see [Note 1](#note1)). In practice, however, this greatly complicates + debugging a faulty node, since the only way to respond to errors is to panic + the node which loses valuable context that could have been logged. + +- There are subtle concurrency management dependencies between the client and + the server that are not clearly documented anywhere, and it is very easy for + small changes in both the client and the server to lead to tricky deadlocks, + panics, race conditions, and slowdowns. As a recent example of this, see + . + +These limitations are fixable, but one important question is whether it is +worthwhile to fix them. We can add request and method identifiers, for +example, but doing so would be a breaking change to the protocol requiring +every application using it to update. If applications have to migrate anyway, +the stability and language coverage of gRPC have improved a lot, and today it +is probably simpler to set up and maintain an application using gRPC transport +than to reimplement the Tendermint socket protocol. + +Moreover, gRPC addresses all the above issues out-of-the-box, and requires +(much) less custom code for both the server (i.e., the application) and the +client. The project is well-funded and widely-used, which makes it a safe bet +for a dependency. + +## Decision + +There is a set of related alternatives to consider: + +- Question 1: Designate a single IPC standard for out-of-process applications? + + Claim: We should converge on one (and only one) IPC option for out-of-process + applications. We should choose an option that, after a suitable period of + deprecation for alternatives, will address most or all the highest-impact + uses of Tendermint. Maintaining multiple options increases the surface area + for bugs and vulnerabilities, and we should not have multiple options for + basic interfaces without a clear and well-documented reason. + +- Question 2a: Choose gRPC and deprecate/remove the socket protocol? + + Claim: Maintaining and improving a custom RPC protocol is a substantial + project and not directly relevant to the requirements of consensus. We would + be better served by depending on a well-maintained open-source library like + gRPC. + +- Question 2b: Improve the socket protocol and deprecate/remove gRPC? + + Claim: If we find meaningful advantages to maintaining our own custom RPC + protocol in Tendermint, we should treat it as a first-class project within + the core and invest in making it good enough that we do not require other + options. + +**One important consideration** when discussing these questions is that _any +outcome which includes keeping the socket protocol will have eventual migration +impacts for out-of-process applications_ regardless. To fix the limitations of +the socket protocol as it is currently designed will require making _breaking +changes_ to the protocol. So, while we may put off a migration cost for +out-of-process applications by retaining the socket protocol in the short term, +we will eventually have to pay those costs to fix the problems in its current +design. + +## Detailed Design + +1. If we choose to standardize on gRPC, the main work in Tendermint core will + be removing and cleaning up the code for the socket client and server. + + Besides the code cleanup, we will also need to clearly document a + deprecation schedule, and invest time in making the migration easier for + applications currently using the socket protocol. + + > **Point for discussion:** Migrating from the socket protocol to gRPC + > should mostly be a plumbing change, as long as we do it during a release + > in which we are not making other breaking changes to ABCI. However, the + > effort may be more or less depending on how gRPC integration works in the + > application's implementation language, and would have to be sure networks + > have plenty of time not only to make the change but to verify that it + > preserves the function of the network. + > + > What questions should we be asking node operators and application + > developers to understand the migration costs better? + +2. If we choose to keep only the socket protocol, we will need to follow up + with a more detailed design for extending and upgrading the protocol to fix + the existing performance and operational issues with the protocol. + + Moreover, since the gRPC interface has been around for a long time we will + also need a deprecation plan for it. + +3. If we choose to keep both options, we will still need to do all the work of + (2), but the gRPC implementation should not require any immediate changes. + + +## Alternatives Considered + +- **FFI**. Another approach we could take is to use a C-based FFI interface so + that applications written in other languages are linked directly with the + consensus node, an option currently only available for Go applications. + + An FFI interface is possible for a lot of languages, but FFI support varies + widely in coverage and quality across languages and the points of friction + can be tricky to work around. Moreover, it's much harder to add FFI support + to a language where it's missing after-the-fact for an application developer. + + Although a basic FFI interface is not too difficult on the Go side, the C + shims for an FFI can get complicated if there's a lot of variability in the + runtime environment on the other end. + + If we want to have one answer for non-Go applications, we are better off + picking an IPC-based solution (whether that's gRPC or an extension of our + custom socket protocol or something else). + +## Consequences + +- **Standardize on gRPC** + + - ✅ Addresses existing performance and operational issues. + - ✅ Replaces custom code with a well-maintained widely-used library. + - ✅ Aligns with Cosmos SDK, which already uses gRPC extensively. + - ✅ Aligns with priv validator interface, for which the socket protocol is already deprecated for gRPC. + - ❓ Applications will be hard to implement in a language without gRPC support. + - ⛔ All users of the socket protocol have to migrate to gRPC, and we believe most current out-of-process applications use the socket protocol. + +- **Standardize on socket protocol** + + - ✅ Less immediate impact for existing users (but see below). + - ✅ Simplifies ABCI API surface by removing gRPC. + - ❓ Users of the socket protocol will have a (smaller) migration. + - ❓ Potentially easier to implement for languages that do not have support. + - ⛔ Need to do all the work to fix the socket protocol (which will require existing users to update anyway later). + - ⛔ Ongoing maintenance burden for per-language server implementations. + +- **Keep both options** + + - ✅ Less immediate impact for existing users (but see below). + - ❓ Users of the socket protocol will have a (smaller) migration. + - ⛔ Still need to do all the work to fix the socket protocol (which will require existing users to update anyway later). + - ⛔ Requires ongoing maintenance and support of both gRPC and socket protocol integrations. + + +## References + +- [Application Blockchain Interface (ABCI)][abci] +- [Tendermint ABCI socket client][socket-client] +- [Tendermint ABCI gRPC client][grpc-client] +- [Initial commit of gRPC client][abci-start] + +[abci]: https://github.com/tendermint/tendermint/tree/main/spec/abci +[socket-client]: https://github.com/tendermint/tendermint/blob/main/abci/client/socket_client.go +[socket-server]: https://github.com/tendermint/tendermint/blob/main/abci/server/socket_server.go +[grpc-client]: https://github.com/tendermint/tendermint/blob/main/abci/client/grpc_client.go +[abci-start]: https://github.com/tendermint/abci/commit/1ab3c747182aaa38418258679c667090c2bb1e0d + +## Notes + +- **Note 1**: The choice to make all ABCI errors protocol-fatal + was intended to avoid the risk that recovering an application error could + cause application state to diverge. Divergence can break consensus, so it's + essential to avoid it. + + This is a sound principle, but conflates protocol errors with "mechanical" + errors such as timeouts, resoures exhaustion, failed connections, and so on. + Because the protocol has no way to distinguish these conditions, the only way + for an application to report an error is to panic or crash. + + Whether a node is running in the same process as the application or as a + separate process, application errors should not be suppressed or hidden. + However, it's important to ensure that errors are handled at a consistent and + well-defined point in the protocol: Having the application panic or crash + rather than reporting an error means the node sees different results + depending on whether the application runs in-process or out-of-process, even + if the application logic is otherwise identical. + +## Appendix: Known Implementations of ABCI Socket Protocol + +This is a list of known implementations of the Tendermint custom socket +protocol. Note that in most cases I have not checked how complete or correct +these implementations are; these are based on search results and a cursory +visual inspection. + +- Tendermint Core (Go): [client][socket-client], [server][socket-server] +- Informal Systems [tendermint-rs](https://github.com/informalsystems/tendermint-rs) (Rust): [client](https://github.com/informalsystems/tendermint-rs/blob/master/abci/src/client.rs), [server](https://github.com/informalsystems/tendermint-rs/blob/master/abci/src/server.rs) +- Tendermint [js-abci](https://github.com/tendermint/js-abci) (JS): [server](https://github.com/tendermint/js-abci/blob/master/src/server.js) +- [Hotmoka](https://github.com/Hotmoka/hotmoka) ABCI (Java): [server](https://github.com/Hotmoka/hotmoka/blob/master/io-hotmoka-tendermint-abci/src/main/java/io/hotmoka/tendermint_abci/Server.java) +- [Tower ABCI](https://github.com/penumbra-zone/tower-abci) (Rust): [server](https://github.com/penumbra-zone/tower-abci/blob/main/src/server.rs) +- [abci-host](https://github.com/datopia/abci-host) (Clojure): [server](https://github.com/datopia/abci-host/blob/master/src/abci/host.clj) +- [abci_server](https://github.com/KrzysiekJ/abci_server) (Erlang): [server](https://github.com/KrzysiekJ/abci_server/blob/master/src/abci_server.erl) +- [py-abci](https://github.com/davebryson/py-abci) (Python): [server](https://github.com/davebryson/py-abci/blob/master/src/abci/server.py) +- [scala-tendermint-server](https://github.com/intechsa/scala-tendermint-server) (Scala): [server](https://github.com/InTechSA/scala-tendermint-server/blob/master/src/main/scala/lu/intech/tendermint/Server.scala) +- [kepler](https://github.com/f-o-a-m/kepler) (Rust): [server](https://github.com/f-o-a-m/kepler/blob/master/hs-abci-server/src/Network/ABCI/Server.hs) diff --git a/cometbft/v0.38/docs/rfc/tendermint-core/rfc-023-semi-permanent-testnet.md b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-023-semi-permanent-testnet.md new file mode 100644 index 00000000..7f5e68cd --- /dev/null +++ b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-023-semi-permanent-testnet.md @@ -0,0 +1,265 @@ +# RFC 023: Semi-permanent Testnet + +## Changelog + +- 2022-07-28: Initial draft (@mark-rushakoff) +- 2022-07-29: Renumber to 023, minor clarifications (@mark-rushakoff) + +## Abstract + +This RFC discusses a long-lived testnet, owned and operated by the Tendermint engineers. +By owning and operating a production-like testnet, +the team who develops Tendermint becomes more capable of discovering bugs that +only arise in production-like environments. +They also build expertise in operating Tendermint; +this will help guide the development of Tendermint towards operator-friendly design. + +The RFC details a rough roadmap towards a semi-permanent testnet, some of the considered tradeoffs, +and the expected outcomes from following this roadmap. + +## Background + +The author's understanding -- which is limited as a new contributor to the Tendermint project -- +is that Tendermint development has been largely treated as a library for other projects to consume. +Of course effort has been spent on unit tests, end-to-end tests, and integration tests. +But whether developing a library or an application, +there is no substitute for putting the software under a production-like load. + +First, there are classes of bugs that are unrealistic to discover in environments +that do not resemble production. +But perhaps more importantly, there are "operational features" that are best designed +by the authors of a given piece of software. +For instance, does the software have sufficient observability built-in? +Are the reported metrics useful? +Are the log messages clear and sufficiently detailed, without being too noisy? + +Furthermore, if the library authors are not only building -- +but also maintaining and operating -- an application built on top of their library, +the authors will have a greatly increased confidence that their library's API +is appropriate for other application authors. + +Once the decision has been made to run and operate a service, +one of the next strategic questions is that of deploying said service. +The author strongly holds the opinion that, when possible, +a continuous delivery model offers the most compelling set of advantages: + +- The code on a particular branch (likely `main` or `master`) is exactly what is, + or what will very soon be, running in production +- There are no manual steps involved in deploying -- other than merging your pull request, + which you had to do anyway +- A bug discovered in production can be rapidly confirmed as fixed in production + +In summary, if the tendermint authors build, maintain, and continuously deliver an application +intended to serve as a long-lived testnet, they will be able to state with confidence: + +- We operate the software in a production-like environment and we have observed it to be + stable and performant to our requirements +- We have discovered issues in production before any external parties have consumed our software, + and we have addressed said issues +- We have successfully used the observability tooling built into our software + (perhaps in conjunction with other off-the-shelf tooling) + to diagnose and debug issues in production + +## Discussion + +The Discussion Section proposes a variety of aspects of maintaining a testnet for Tendermint. + +### Number of testnets + +There should probably be one testnet per maintained branch of Tendermint, +i.e. one for the `main` branch +and one per `v0.N.x` branch that the authors maintain. + +There may also exist testnets for long-lived feature branches. + +We may eventually discover that there is good reason to run more than one testnet for a branch, +perhaps due to a significant configuration variation. + +### Testnet lifecycle + +The document has used the terms "long-lived" and "semi-permanent" somewhat interchangeably. +The intent of the testnet being discussed in this RFC is to exist indefinitely; +but there is a practical understanding that there will be testnet instances +which will be retired due to a variety of reasons. +For instance, once a release branch is no longer supported, +its corresponding testnet should be torn down. + +In general, new commits to branches with corresponding testnets +should result in an in-place upgrade of all nodes in the testnet +without any data loss and without requiring new configuration. +The mechanism for achieving this is outside the scope of this RFC. + +However, it is also expected that there will be +breaking changes during the development of the `main` branch. +For instance, suppose there is an unreleased feature involving storage on disk, +and the developers need to change the storage format. +It should be at the developers' discretion whether it is feasible and worthwhile +to introduce an intermediate commit that translates the old format to the new format, +or if it would be preferable to just destroy the testnet and start from scratch +without any data in the old format. + +Similarly, if a developer inadvertently pushed a breaking change to an unreleased feature, +they are free to make a judgement call between reverting the change, +adding a commit to allow a forward migration, +or simply forcing the testnet to recreate. + +### Testnet maintenance investment + +While there is certainly engineering effort required to build the tooling and infrastructure +to get the testnets up and running, +the intent is that a running testnet requires no manual upkeep under normal conditions. + +It is expected that a subset of the Tendermint engineers are familiar with and engaged in +writing the software to maintain and build the testnet infrastructure, +but the rest of the team should not need any involvement in authoring that code. + +The testnets should be configured to send notifications for events requiring triage, +such as a chain halt or a node OOMing. +The time investment necessary to address the underlying issues for those kind of events +is unpredictable. + +Aside from triaging exceptional events, an engineer may choose to spend some time +collecting metrics or profiles from testnet nodes to check performance details +before and after a particular change; +or they may inspect logs associated with an expected behavior change. +But during day-to-day work, engineers are not expected to spend any considerable time +directly interacting with the testnets. + +If we discover that there are any routine actions engineers must take against the testnet +that take any substantial focused time, +those actions should be automated to a one-line command as much as is reasonable. + +### Testnet MVP + +The minimum viable testnet meets this set of features: + +- The testnet self-updates following a new commit pushed to Tendermint's `main` branch on GitHub + (there are some omitted steps here, such as CI building appropriate binaries and + somehow notifying the testnet that a new build is available) +- The testnet runs the Tendermint KV store for MVP +- The testnet operators are notified if: + - Any node's process exits for any reason other than a restart for a new binary + - Any node stops updating blocks, and by extension if a chain halt occurs + - No other observability will be considered for MVP +- The testnet has a minimum of 1 full node and 3 validators +- The testnet has a reasonably low, constant throughput of transactions -- say 30 tx/min -- + and the testnet operators are notified if that throughput drops below 75% of target + sustained over 5 minutes +- The testnet only needs to run in a single datacenter/cloud-region for MVP, + i.e. running in multiple datacenters is out of scope for MVP +- The testnet is running directly on VMs or compute instances; + while Kubernetes or other orchestration frameworks may offer many significant advantages, + the Tendermint engineers should not be required to learn those tools in order to + perform basic debugging + +### Testnet medium-term goals + +The medium-term goals are intended to be achievable within the 6-12 month time range +following the launch of MVP. +These goals could realistically be roadmapped following the launch of the MVP testnet. + +- The `main` testnet has more than 20 nodes (completely arbitrary -- 5x more than 1+3 at MVP) +- In addition to the `main` testnet, + there is at least one testnet associated with one release branch +- The testnet no longer is simply running the Tendermint KV store; + now it is built on a more complex, custom application + that deliberately exercises a greater portion of the Tendermint stack +- Each testnet is spread across at least two cloud providers, + in order to communicate over a network more closely resembling use of Tendermint in "real" chains +- The node updates have some "jitter", + with some nodes updating immediately when a new build is available, + and others delaying up to perhaps 30-60 minutes +- The team has published some form of dashboards that have served well for debugging, + which external parties can copy/modify to their needs + - The dashboards must include metrics published by Tendermint nodes; + there should be both OS- or runtime-level metrics such as memory in use, + and application-level metrics related to the underlying blockchain + - "Published" in this context is more in the spirit of "shared with the community", + not "produced a supported open source tool" -- + this could be published to GitHub with a warning that no support is offered, + or it could simply be a blog post detailing what has worked for the Tendermint developers + - The dashboards will likely be implemented on free and open source tooling, + but that is not a hard requirement if paid software is more appropriate +- The team has produced a reference model of a log aggregation stack that external parties can use + - Similar to the "published" dashboards, this only needs to be "shared" rather than "supported" +- Chaos engineering has begun being integrated into the testnets + (this could be periodic CPU limiting or deliberate network interference, etc. + but it probably would not be filesystem corruption) +- Each testnet has at least one node running a build with the Go race detector enabled +- The testnet contains some kind of generalized notification system built in: + - Tendermint code grows "watchdog" systems built in to validate things like + subsystems have not deadlocked; e.g. if the watchdog can't acquire and immediately release + a particular mutex once in every 5-minute period, it is near certain that the target + subsystem has deadlocked, and an alert must be sent to the engineering team. + (Outside of the testnet, the watchdogs could be disabled, or they could panic on failure.) + - The notification system does some deduplication to minimize spam on system failure + +### Testnet long-term vision + +The long-term vision includes goals that are not necessary for short- or medium-term success, +but which would support building an increasingly stable and performant product. +These goals would generally be beyond the one-year plan, +and therefore they would not be part of initial planning. + +- There is a centralized dashboard to get a quick overview of all testnets, + or at least one centralized dashboard per testnet, + showing TBD basic information +- Testnets include cloud spot instances which periodically and abruptly join and leave the network +- The testnets are a heterogeneous mixture of straight VMs and Docker containers, + thereby more closely representing production blockchains +- Testnets have some manner of continuous profiling, + so that we can produce an apples-to-apples comparison of CPU/memory cost of particular operations + +### Testnet non-goals + +There are some things we are explicitly not trying to achieve with long-lived testnets: + +- The Tendermint engineers will NOT be responsible for the testnets' availability + outside of working hours; there will not be any kind of on-call schedule +- As a result of the 8x5 support noted in the previous point, + there will be NO guarantee of uptime or availability for any testnet +- The testnets will NOT be used to gate pull requests; + that responsibility belongs to unit tests, end-to-end tests, and integration tests +- Similarly, the testnet will NOT be used to automate any changes back into Tendermint source code; + we will not automatically create a revert commit due to a failed rollout, for instance +- The testnets are NOT intended to have participation from machines outside of the + Tendermint engineering team's control, as the Tendermint engineers are expected + to have full access to any instance where they may need to debug an issue +- While there will certainly be individuals within the Tendermint engineering team + who will continue to build out their individual "devops" skills to produce + the infrastructure for the testnet, it is NOT a goal that every Tendermint engineer + is even _familiar_ with the tech stack involved, whether it is Ansible, Terraform, + Kubernetes, etc. + As a rule of thumb, all engineers should be able to get shell access on any given instance + and should have access to the instance's logs. + Little if any further operational skills will be expected. +- The testnets are not intended to be _created_ for one-off experiments. + While there is nothing wrong with an engineer directly interacting with a testnet + to try something out, + a testnet comes with a considerable amount of "baggage", so end-to-end or integration tests + are closer to the intent for "trying something to see what happens". + Direct interaction should be limited to standard blockchain operations, + _not_ modifying configuration of nodes. +- Likewise, the purpose of the testnet is not to run specific "tests" per se, + but rather to demonstrate that Tendermint blockchains as a whole are stable + under a production load. + Of course we will inject faults periodically, but the intent is to observe and prove that + the testnet is resilient to those faults. + It would be the responsibility of a lower-level test to demonstrate e.g. + that the network continues when a single validator disappears without warning. +- The testnet descriptions in this document are scoped only to building directly on Tendermint; + integrating with the Cosmos SDK, or any other third-party library, is out of scope + +### Team outcomes as a result of maintaining and operating a testnet + +Finally, this section reiterates what team growth we expect by running semi-permanent testnets. + +- Confidence that Tendermint is stable under a particular production-like load +- Familiarity with typical production behavior of Tendermint, e.g. what the logs look like, + what the memory footprint looks like, and what kind of throughput is reasonable + for a network of a particular size +- Comfort and familiarity in manually inspecting a misbehaving or failing node +- Confidence that Tendermint ships sufficient tooling for external users + to operate their nodes +- Confidence that Tendermint exposes useful metrics, and comfort interpreting those metrics +- Produce useful reference documentation that gives operators confidence to run Tendermint nodes diff --git a/cometbft/v0.38/docs/rfc/tendermint-core/rfc-024-block-structure-consolidation.md b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-024-block-structure-consolidation.md new file mode 100644 index 00000000..91dec2d6 --- /dev/null +++ b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-024-block-structure-consolidation.md @@ -0,0 +1,362 @@ +# RFC 024: Block Structure Consolidation + +## Changelog + +- 19-Apr-2022: Initial draft started (@williambanfield). +- 3-May-2022: Initial draft complete (@williambanfield). + +## Abstract + +The `Block` data structure is a very central structure within Tendermint. Because +of its centrality, it has gained several fields over the years through accretion. +Not all of these fields may be necessary any more. This document examines which +of these fields may no longer be necessary for inclusion in the block and makes +recommendations about how to proceed with each of them. + +## Background + +The current block structure contains multiple fields that are not required for +validation or execution of a Tendermint block. Some of these fields had vestigial +purposes that they no longer serve and some of these fields exist as a result of +internal Tendermint domain objects leaking out into the external data structure. + +In so far as is possible, we should consolidate and prune these superfluous +fields before releasing a 1.0 version of Tendermint. All pruning of these +fields should be done with the aim of simplifying the structures to what +is needed while preserving information that aids with debugging and that also +allow external protocols to function more efficiently than if they were removed. + +### Current Block Structure + +The current block structures are included here to aid discussion. + +```proto +message Block { + Header header = 1; + Data data = 2; + tendermint.types.EvidenceList evidence = 3; + Commit last_commit = 4; +} +``` + +```proto +message Header { + tendermint.version.Consensus version = 1; + string chain_id = 2; + int64 height = 3; + google.protobuf.Timestamp time = 4; + BlockID last_block_id = 5; + bytes last_commit_hash = 6; + bytes data_hash = 7; + bytes validators_hash = 8; + bytes next_validators_hash = 9; + bytes consensus_hash = 10; + bytes app_hash = 11; + bytes last_results_hash = 12; + bytes evidence_hash = 13; + bytes proposer_address = 14; +} + +``` + +```proto +message Data { + repeated bytes txs = 1; +} +``` + +```proto +message EvidenceList { + repeated Evidence evidence = 1; +} +``` + +```proto +message Commit { + int64 height = 1; + int32 round = 2; + BlockID block_id = 3; + repeated CommitSig signatures = 4; +} +``` + +```proto +message CommitSig { + BlockIDFlag block_id_flag = 1; + bytes validator_address = 2; + google.protobuf.Timestamp timestamp = 3; + bytes signature = 4; +} +``` + +```proto +message BlockID { + bytes hash = 1; + PartSetHeader part_set_header = 2; +} +``` + +### On Tendermint Blocks + +#### What is a Tendermint 'Block'? + +A block is the structure produced as the result of an instance of the Tendermint +consensus algorithm. At its simplest, the 'block' can be represented as a Merkle +root hash of all of the data used to construct and produce the hash. Our current +block proto structure includes _far_ from all of the data used to produce the +hashes included in the block. + +It does not contain the full `AppState`, it does not contain the `ConsensusParams`, +nor the `LastResults`, nor the `ValidatorSet`. Additionally, the layout of +the block structure is not inherently tied to this Merkle root hash. Different +layouts of the same set of data could trivially be used to construct the +exact same hash. The thing we currently call the 'Block' is really just a view +into a subset of the data used to construct the root hash. Sections of the +structure can be modified as long as alternative methods exist to query and +retrieve the constituent values. + +#### Why this digression? + +This digression is aimed at informing what it means to consolidate 'fields' in the +'block'. The discussion of what should be included in the block can be teased +apart into a few different lines of inquiry. + +1. What values need to be included as part of the Merkle tree so that the +consensus algorithm can use proof-of-stake consensus to validate all of the +properties of the chain that we would like? +2. How can we create views of the data that can be easily retrieved, stored, and +verified by the relevant protocols? + +These two concerns are intertwined at the moment as a result of how we store +and propagate our data but they don't necessarily need to be. This document +focuses primarily on the first concern by suggesting fields that can be +completely removed without any loss in the function of our consensus algorithm. + +This document also suggests ways that we may update our storage and propagation +mechanisms to better take advantage of Merkle tree nature of our data although +these are not its primary concern. + +## Discussion + +### Data to consider removing + +This section proposes a list of data that could be completely removed from the +Merkle tree with no loss to the functionality of our consensus algorithm. + +Where the change is possible but would hamper external protocols or make +debugging more difficult, that is noted in discussion. + +#### CommitSig.Timestamp + +This field contains the timestamp included in the precommit message that was +issued for the block. The field was once used to produce the timestamp of the block. +With the introduction of Proposer-Based Timestamps, This field is no longer used +in any Tendermint algorithms and can be completely removed. + +#### CommitSig.ValidatorAddress + +The `ValidatorAddress` is included in each `CommitSig` structure. This field +is hashed along with all of the other fields of the `CommitSig`s in the block +to form the `LastCommitHash` field in the `Header`. The `ValidatorAddress` is +somewhat redundant in the hash. Each validator has a unique position in the +`CommitSig` and the hash is built preserving this order. Therefore, the +information of which signature corresponds to which validator is included in +the root hash, even if the address is absent. + +It's worth noting that the validator address could still be included in the +_hash_ even if it is absent from the `CommitSig` structure in the block by +simply hashing it locally at each validator but not including it in the block. +The reverse is also true. It would be perfectly possible to not include the +`ValidatorAddress` data in the `LastCommitHash` but still include the field in +the block. + +#### BlockID.PartSetHeader + +The [BlockID][block-id] field comprises the [PartSetHeader][part-set-header] and the hash of the block. +The `PartSetHeader` is used by nodes to gossip the block by dividing it into +parts. Nodes receive the `PartSetHeader` from their peers, informing them of +what pieces of the block to gather. There is no strong reason to include this +value in the block. Validators will still be able to gossip and validate the +blocks that they received from their peers using this mechanism even if it is +not written into the block. The `BlockID` can therefore be consolidated into +just the hash of the block. This is by far the most uncontroversial change +and there appears to be no good reason _not_ to do it. Further evidence that +the field is not meaningful can be found in the fact that the field is not +actually validated to ensure it is correct during block validation. Validation +only checks that the [field is well formed][psh-check]. + +#### ChainID + +The `ChainID` is a string selected by the chain operators, usually a +human-readable name for the network. This value is immutable for the lifetime +of the chain and is defined in the genesis file. It is therefore hashed into the +original block and therefore transitively included as in the Merkle root hash of +every block. The redundant field is a candidate for removal from the root hash +of each block. However, aesthetically, it's somewhat nice to include in each +block, as if the block was 'stamped' with the ID. Additionally, re-validating +the value from genesis would be painful and require reconstituting potentially +large chains. I'm therefore mildly in favor of maintaining this redundant +piece of information. We pay almost no storage cost for maintaining this +identical data, so the only cost is in the time required to hash it into the +structure. + +#### LastResultsHash + +`LastResultsHash` is a hash covering the result of executing the transactions +from the previous block. It covers the response `Code`, `Data`, `GasWanted`, +and `GasUsed` with the aim of ensuring that execution of all of the transactions +was performed identically on each node. The data covered by this field _should_ +be also reflected in the `AppHash`. The `AppHash` is provided by the application +and should be deterministically calculated by each node. This field could +therefore be removed on the grounds that its data is already reflected elsewhere. + +I would advocate for keeping this field. This field provides an additional check +for determinism across nodes. Logic to update the application hash is more +complicated for developers to implement because it relies either on building a +complete view of the state of the application data. The `Results` returned by +the application contain simple response codes and deterministic data bytes. +Leaving the field will allow for transaction execution issues that are not +correctly reflected in the `AppHash` to be more completely diagnosed. + +Take the case of mismatch of `LastResultsHash` between two nodes, A and B, where both +nodes report divergent values. If `A` and `B` both report +the same `AppHash`, then some non-deterministic behavior occurred that was not +accurately reflected in the `AppHash`. The issue caused by this +non-determinism may not show itself for several blocks, but catching the divergent +state earlier will improve the chances that a chain is able to recover. + +#### ValidatorsHash + +Both `ValidatorsHash` and `NextValidatorsHash` are included in the block +header. `Validatorshash` contains the hash of the [public key and voting power][val-hash] +of each validator in the active set for the current block and `NextValidatorsHash` +contains the same data but for the next height. + +This data is effectively redundant. Having both values present in the block +structure is helpful for light client verification. The light client is able to +easily determine if two sequential blocks used the same validator set by querying +only one header. + +`ValidatorsHash` is also important to the light client algorithm for performing block +validation. The light client uses this field to ensure that the validator set +it fetched from a full node is correct. It can be sure of the correctness of +the retrieved structure by hashing it and checking the hash against the `ValidatorsHash` +of the block it is verifying. Because a validator that the light client trusts +signed over the `ValidatorsHash`, it can be certain of the validity of the +structure. Without this check, phony validator sets could be handed to the light +client and the code tricked into believing a different validator set was present +at a height, opening up a major hole in the light client security model. + +This creates a recursive problem. To verify the validator set that signed the +block at height `H`, what information do we need? We could fetch the +`NextValidatorsHash` from height `H-1`, but how do we verify that that hash is correct? + +#### ProposerAddress + +The section below details a change to allow the `ProposerAddress` to be calculated +from a field added to the block. This would allow the `Address` to be dropped +from the block. Consumers of the chain could run the proposer selection [algorithm][proposer-selection] +to determine who proposed each block. + +I would advocate against this. Any consumer of the chain that wanted to +know which validator proposed a block would have to run the proposer selection +algorithm. This algorithm is not widely implemented, meaning that consumers +in other languages would need to implement the algorithm to determine a piece +of basic information about the chain. + +### Data to consider adding + +#### ProofOfLockRound + +The _proof of lock round_ is the round of consensus for a height in which the +Tendermint algorithm observed a super majority of voting power on the network for +a block. + +Including this value in the block will allow validation of currently +un-validated metadata. Specifically, including this value will allow Tendermint +to validate that the `ProposerAddress` in the block is correct. Without knowing +the locked round number, Tendermint cannot calculate which validator was supposed +to propose a height. Because of this, our [validation logic][proposer-check] does not check that +the `ProposerAddress` included in the block corresponds to the validator that +proposed the height. Instead, the validation logic simply checks that the value +is an address of one of the known validators. + +Currently, we maintain the _committed round_ in the `Commit` for height `H`, which is +written into the block at height `H+1`. This value corresponds to the round in +which the proposer of height `H+1` received the commit for height `H`. The proof +of lock round would not subsume this value. + +### Additional possible updates + +#### Updates to storage + +Currently we store the [every piece of each block][save-block] in the `BlockStore`. +I suspect that this has lead to some mistakes in reasoning around the merits of +consolidating fields in the block. We could update the storage scheme we use to +store only some pieces of each block and still achieve a space savings without having +to change the block structure at all. + +The main way to achieve this would be by _no longer saving data that does not change_. +At each height we save a set of data that is unlikely to have changed from the +previous height in the block structure, this includes the `ValidatorAddress`es, +the `ValidatorsHash`, the `ChainID`. These do not need to be saved along with +_each_ block. We could easily save the value and the height at which the value +was updated and construct each block using the data that existed at the time. + +This document does not make any specific recommendations around storage since +that is likely to change with upcoming improvements to to the database infrastructure. +However, it's important to note that removing fields from the block for the +purposes of 'saving space' may not be that meaningful. We should instead focus +our attention of removing fields from the block that are no longer needed +for correct functioning of the protocol. + +#### Updates to propagation + +Block propagation suffers from the same issue that plagues block storage, we +propagate all of the contents of each block proto _even when these contents are redundant +or unchanged from previous blocks_. For example, we propagate the `ValidatorAddress`es +for each block in the `CommitSig` structure even when it never changed from a +previous height. We could achieve a speed-up in many cases by communicating the +hashes _first_ and letting peers request additional information when they do not +recognize the communicated hash. + +For example, in the case of the `ValidatorAddress`es, the node would first +communicate the `ValidatorsHash` of the block to its peers. The peers would +check their storage for a validator set matching the provided hash. If the peer +has a matching set, it would populate its local block structure with the +appropriate values from its store. If peer did not have a matching set, it would +issue a request to its peers, either via P2P or RPC for the data it did not have. + +Conceptually, this is very similar to how content addressing works in protocols +such as git where pushing a commit does not require pushing the entire contents +of the tree referenced by the commit. + +### Impact on light clients + +As outlined in the section [On Tendermint Blocks](#on-tendermint-blocks), there +is a distinction between what data is referenced in the Merkle root hash and the +contents of the proto structure we currently call the `Block`. + +Any changes to the Merkle root hash will necessarily be breaking for legacy light clients. +Either a soft-upgrades scheme will need to be implemented or a hard fork will +be required for chains and light clients to function with the new hashes. +This means that all of the additions and deletions from the Merkle root hash +proposed by this document will be light client breaking. + +Changes to the block structure alone are not necessarily light client breaking if the +data being hashed are identical and legacy views into the data are provided +for old light clients during transitions. For example, a newer version of the +block structure could move the `ValidatorAddress` field to a different field +in the block while still including it in the hashed data of the `LastCommitHash`. +As long as old light clients could still fetch the old data structure, then +this would not be light client breaking. + +## References + +[part-set-header]: https://github.com/tendermint/tendermint/blob/208a15dadf01e4e493c187d8c04a55a61758c3cc/types/part_set.go#L94 +[block-id]: https://github.com/tendermint/tendermint/blob/208a15dadf01e4e493c187d8c04a55a61758c3cc/types/block.go#L1090 +[psh-check]: https://github.com/tendermint/tendermint/blob/208a15dadf01e4e493c187d8c04a55a61758c3cc/types/part_set.go#L116 +[proposer-selection]: https://github.com/tendermint/tendermint/blob/208a15dadf01e4e493c187d8c04a55a61758c3cc/spec/consensus/proposer-selection.md +[val-hash]: https://github.com/tendermint/tendermint/blob/29e5fbcc648510e4763bd0af0b461aed92c21f30/types/validator.go#L160 +[proposer-check]: https://github.com/tendermint/tendermint/blob/29e5fbcc648510e4763bd0af0b461aed92c21f30/internal/state/validation.go#L102 +[save-block]: https://github.com/tendermint/tendermint/blob/59f0236b845c83009bffa62ed44053b04370b8a9/internal/store/store.go#L490 diff --git a/cometbft/v0.38/docs/rfc/tendermint-core/rfc-025-support-app-side-mempool.md b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-025-support-app-side-mempool.md new file mode 100644 index 00000000..63dea869 --- /dev/null +++ b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-025-support-app-side-mempool.md @@ -0,0 +1,300 @@ +# RFC 25: Support Application Defined Transaction Storage (app-side mempools) + +## Changelog + +- Aug 17, 2022: initial draft (@williambanfield) +- Aug 19, 2022: updated draft (@williambanfield) + +## Abstract + +With the release of ABCI++, specifically the `PrepareProposal` call, the utility +of the Tendermint mempool becomes much less clear. This RFC discusses possible +changes that should be considered to Tendermint to better support applications +that intend to use `PrepareProposal` to implement much more powerful transaction +ordering and filtering functionality than Tendermint can provide. It proposes +scoping down the responsibilities of Tendermint to suit this new use case. + +## Background + +Tendermint currently ships with a data structure it calls the +[mempool][mempool-link]. The mempool's primary function is to store pending +valid transactions. Tendermint uses the contents of the mempool in two main +ways: 1) to gossip these pending transactions to other nodes on a Tendermint +network and 2) to select transactions to be included in a proposed block. Before +ABCI++, when proposing a block Tendermint selects the next set of transactions +from the mempool that fit within block and proposes them. + +There are a few issues with this data structure. These include issues of how +transaction validity is defined, how transactions should be ordered and selected +for inclusion in the next block, and when a transaction should start or stop +being gossiped. The creation of `PrepareProposal` in ABCI++ adds the additional +issue of unclear ownership over which entity, Tendermint or the ABCI +application, is responsible for selecting the transactions to be included in +a proposed block. + +None of these issues of validity, ordering, and gossiping having simple, +one-size fits all solutions. Different applications will have different +preferences and needs for each of them. The current Tendermint mempool attempts +to strike a balance but is quite prescriptive about these questions. We can +better support a varied range of applications by simplifying the current mempool +and by reducing and clarifying its scope of responsibilities. + +## Discussion + +### The mempool is a leaky abstraction and handles too many concerns + +The current mempool is a leaky abstraction. Presently, Tendermint's mempool keeps +track of a multitude of details that primarily service concerns of the application. + +#### Gas + +The mempool keeps track of Gas, a proxy for how computationally expensive it +will be to execute a transaction. As discussed in [RFC-011](https://github.com/tendermint/tendermint/blob/2313f358003d0c4d9d0e7705b4632d819dfb0d92/docs/rfc/rfc-011-delete-gas.md), this metadata is +not a concern of Tendermint's. Tendermint does not execute transactions. This +data is stored within Tendermint's mempool along with the maximum gas the application +will permit to be used in a block so that Tendermint's mempool can enforce +transaction validity using it: transactions that exceed the configured maximum +are rejected from the mempool. How much 'Gas' a transaction consumes and if that +precludes it from execution by the application is a validity condition imposed +by the application, not Tendermint. It is an application abstraction that leaks +into the Tendermint mempool. + +#### Sender + +The Tendermint mempool stores a `sender` string metadata for each transaction +it receives. The mempool only stores one transaction per sender at any time. +The `sender` metadata is populated by the application during `CheckTx`. +`Sender` uniqueness is enforced separately on each node's mempool. Nothing +prevents multiple transactions with the same `sender` from existing in separate +mempools on the network. + +While multiple transactions from the same sender on a network is a shortcoming +of the `sender` abstraction, the issue posed by sender to the mempool is that +`sender` uniqueness is a condition of transaction validity that is otherwise +meaningless to Tendermint. The `sender` field allows the application to +influence which transactions Tendermint will include next in a block. However, +with the advent of `PrepareProposal`, the application can select directly and +this `sender` field is of marginal benefit. Additionally, applications require +much more expressive validity conditions than just `sender` uniqueness. + +#### Adding additional semantics to the mempool + +The Tendermint mempool is relied upon by every ABCI application. Changing its +code to incorporate new features or update its behavior affects all of +Tendermint's downstream consumers. New applications frequently need ways of +sorting pending transactions and imposing transaction validity conditions. This +data structure cannot change quickly to meet the shifting needs of new +consumers while also maintaining a stable API for the applications that are +already successfully running on top of Tendermint. New strategies for sorting +and validating pending transactions would be best implemented outside of +Tendermint, where creating new semantics does not risk disrupting the existing +users. + +### Tendermint's scope of responsibility + +#### What should Tendermint be responsible for? + +Tendermint's responsibilities should be as narrowly scoped as possible to allow +the code base to be useful for many developers and maintainable by the core +team. + +The Tendermint node maintains a P2P network over which pending transactions, +proposed blocks, votes and other messages are sent. Tendermint, using these +messages, comes to consensus on the proposed blocks and delivers their contents +to the application in an orderly fashion. + +In this description of Tendermint, its only responsibility, in terms of pending +transactions, is to _gossip_ them over its P2P network. Any additional logic +surrounding validity, ordering etc. requires an understanding of the meaning of +the transaction that Tendermint does not and _should not_ have. + +#### What should the application be responsible for? + +Transaction contents have semantic meaning to the ABCI application. Pending +transactions are valid and have execution priority in relationship to the +current state of application. While Tendermint is clearly responsible for the +action of gossiping the transaction, it cannot decide when to start or stop +gossiping any given transaction. While only valid transactions should be +gossiped, as stated, it cannot appropriately make decisions about transaction +validity beyond simple heuristics. The application therefore should be +responsible for defining pending transaction validity, determining when to start +or stop gossiping a transaction, and for selecting which transaction should be +contained within a block. + +### How can Tendermint best be designed for this responsibility? + +With the understanding that Tendermint's responsibility is to gossip the set of +transactions that the application currently considers valid and high priority, +we can update its API and data structures accordingly. With the creation of +`PrepareProposal`, the mempool may be able to drop its responsibility to select +transactions for a block; It can be primarily responsible for gossiping and +nothing else. + +#### Goodbye mempool, hello GossipList + +The mempool contains many structures to retain, order, and select the set of +transactions to gossip and to propose. These mempool structures could be +completely replaced with a single list that allows Tendermint to fulfill the +previously stated responsibility. This proposed list, the `GossipList`, would +simply contain the set of transactions that Tendermint is responsible for +gossiping at the moment. This `GossipList` would be updated by the application +at a set of defined junctures and Tendermint would never add to it or remove +from it without input from the application. Tendermint would impose _no_ +validity constraints on the contents of this list and would not attempt to +remove items unless instructed to. + +### Mock API of the GossipList + +Outlined below is a proposed API for this data structure. These calls would be +added to the ABCI API and would come to replace the current `CheckTx` call. + +#### `OfferPendingTransaction` + +`OfferPendingTransaction` replaces the `CheckTx` call that is invoked when +Tendermint receives a submitted or gossiped transaction. The `GossipList` will +invoke `OfferPendingTransaction` on _every_ transaction sent to Tendermint that +does not match one of the transactions already in the `GossipList`. The mempool +currently drops gossiped transactions before `CheckTx` is called if the +transaction is considered invalid for a Tendermint-defined reason such as +inclusion in the mempool 'cache' or it overflows the max transaction size. + +The application can indicate if the transaction should be added to the +`GossipList` via `ResponseOfferPendingTransaction`'s `GossipStatus` field. If +the `GossipList` is full, the application must list a transaction to remove from +`GossipList`, otherwise the transaction will not be added. In this way, +a transaction will _never_ leave the list unless the application removes it from +the list explicitly. + +```proto +message RequestOfferPendingTransaction { + bytes tx = 1; + int64 gossip_list_max_size = 2; + int64 gossip_list_current_size = 3; +} + +message ResponseOfferPendingTransaction { + GossipStatus gossip_status = 1; + enum GossipStatus { + UNKNOWN = 0; + GOSSIP = 1; + NO_GOSSIP = 2; + } + repeated bytes removals =2 +} +``` + +#### `UpdateTransactionGossipList` + +`UpdateTransactionGossipList` would be a simple method that allows the +application to exactly set the contents of the `GossipList`. Tendermint would +call `UpdateTransactionGossipList` on the application, which would respond with +the list of all transactions to gossip. The contents of the `GossipList` would +be completely replaced with the contents provided by the application in +`UpdateTransactionGossipList`. + +```proto +message UpdateTransactionGossipListRequest { + int64 max_size = 1; // application cannot provide more than `max_size` transactions. +} + +message UpdateTransactionGossipListResponse { + repeated bytes = 1; +} +``` + +This new `ABCI` method would serve multiple functions. First, it would replace +the re-`CheckTx` calls made by Tendermint after each block is committed. After +each block is committed, Tendermint currently passes the entire contents of the +mempool to the application one-by-one via `CheckTx` calls with `CheckTxType` set +to `RECHECK`. The application, in this way, can then inspect the entire mempool +and remove any transactions that became invalid as a result of the most recent +block being committed. + +`UpdateTransactionGossipList` would completely replace this set of re-`CheckTx` +calls. After each block is committed, Tendermint would call +`UpdateTransactionGossipList` and the application would be responsible for +exactly providing the set of transactions for Tendermint to maintain. The IPC +overhead here would be roughly equivalent to the re-`CheckTx` overhead, as the +entire contents of the gossip structure is communicated, but, in the +`UpdateTransactionGossipList` call, the application sends transactions instead +of Tendermint. + +This new method would _also_ replace the mempool's `Update` API. The `Update` +method on the mempool receives the list of transactions that were just executed +as part of the most recent height and removes them from the mempool. The +`GossipList` would have no such method and instead, the application would become +responsible for setting the contents after each block via +`UpdateTransactionGossipList`. This gives the application more control over when +to start and stop gossiping transactions than it has at the moment. In this +call, the application can completely replace the `GossipList`. + +This also complements the `PrepareProposal` call nicely, because a transaction +introduced via `PrepareProposal` may be semantically equivalent to a transaction +present in Tendermint's mempool in a way that Tendermint cannot detect. The +mempool `Update` call only compares transaction hashes, +`UpdateTransactionGossipList` allows the application to easily compare on +transaction contents as well. + +As a nice benefit, it also allows the application to easily continue gossiping +of a transaction that was just executed in the block. Applications may wish to +execute the same transaction multiple times, which the mempool `Update` call +makes very cumbersome by clearing transactions that have the same contents of +those that were just executed. + +### Tendermint startup + +On Tendermint startup, the `GossipList` would be completely empty. It does not +persist transactions and is an in-memory only data structure. To populate the +`GossipList` on startup, Tendermint will issue an `UpdateTransactionGossipList` +call to the application to request the application provide it with a list of +transactions to fill the gossip list. + +### Additional benefits of this API + +#### No more confusing mempool cache + +The current Tendermint mempool stores a [cache][cache-when-clear] of transaction +hashes that should not be accepted into the mempool. When a transaction is sent +to the mempool but is present in the cache the transaction is dropped without +ever being sent to the application via `CheckTx`. This cache is intended to help +the application guard against receiving the same invalid transaction over and +over. However, this means that presence or absence from the mempool cache +becomes a condition of validity for pending transactions. + +Being placed in this cache has serious consequences for a proposed transaction, +but the rules for when a transaction should be placed in this cache are unclear. +So unclear in fact, that conditions for when to include a transaction in this +cache have been completely reversed by different commits +([1][update-remove-from-cache],[2][update-keep-in-cache]) on the Tendermint +project. Additional github issues have noted that it's very ambiguous as to +[when the cache should be cleared][cache-when-clear] and whether or not the +cache should allow [previously invalid transactions][later-valid] to later +become valid. There is no one-size-fits all solution to these problems. +Different applications need very different behavior, so this should ultimately +not be the responsibility of Tendermint. Implementing the `GossipList` clears +Tendermint of this responsibility. + +#### Improved guarantees about the set of transactions being gossiped + +As discussed in the [Mock API](#mock-api-of-the-gossiplist) section, the +`GossipList` only adds and removes or replaces transactions in the `GossipList` +when the application says to. Under this design, the contents of this list are +never ambiguous. The list contains exactly what the application most recently +told Tendermint to gossip, nothing more nothing less. + +### Additional considerations + +This document leaves a few aspects unconsidered that should be understood before +future designs are made in this area: + +1. Impact of duplicating transactions in both the `GossipList` and within the + application. +2. Transition plan and feasibility of migrating applications to the new API. + +## References + +[cache-when-clear]:https://github.com/tendermint/tendermint/issues/7723 +[update-remove-from-cache]:https://github.com/tendermint/tendermint/pull/233 +[update-keep-in-cache]:https://github.com/tendermint/tendermint/issues/2855 +[later-valid]:https://github.com/tendermint/tendermint/issues/458 +[mempool-link]:https://github.com/tendermint/tendermint/blob/c8302c5fcb7f1ffafdefc5014a26047df1d27c99/mempool/mempool.go#L30 diff --git a/cometbft/v0.38/docs/rfc/tendermint-core/rfc-027-p2p-message-bandwidth-report.md b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-027-p2p-message-bandwidth-report.md new file mode 100644 index 00000000..eaa99cde --- /dev/null +++ b/cometbft/v0.38/docs/rfc/tendermint-core/rfc-027-p2p-message-bandwidth-report.md @@ -0,0 +1,287 @@ +# RFC 27: P2P Message Bandwidth Report + +## Changelog + +- Nov 7, 2022: initial draft (@williambanfield) +- Nov 15, 2022: draft completed (@williambanfield) + +## Abstract + +Node operators and application developers complain that Tendermint nodes consume +larges amounts of network bandwidth. This RFC catalogues the major sources of bandwidth +consumption within Tendermint and suggests modifications to Tendermint that may reduce +bandwidth consumption for nodes. + +## Background +Multiple teams running validators in production report that the validator +consumes a lot of bandwidth. They report that operators running on a network +with hundreds of validators consumes multiple terabytes of bandwidth per day. +Prometheus data collected from a validator node running on the Osmosis chain +shows that Tendermint sends and receives large amounts of data to peers. In the +nearly three hours of observation, Tendermint sent nearly 42 gigabytes and +received about 26 gigabytes, for an estimated 366 gigabytes sent daily and 208 +gigabytes received daily. While this is shy of the reported terabytes number, +operators running multiple nodes for a 'sentry' pattern could easily send and +receive a terabyte of data. + +Sending and receiving large amounts of data has a cost for node operators. Most +cloud platforms charge for network traffic egress. Google Cloud charges between +[$.05 to $.12 per gigabyte of egress traffic][gcloud-pricing], and ingress is +free. Hetzner [charges 1€ per TB used over the 10-20TB base bandwidth per +month][hetzner-pricing], which will be easily hit if multiple terabytes are +sent and received per day. Using the values collected from the validator on +Osmosis, a single node on Google cloud may cost $18 to $44 a day running on +Google cloud. On Hetzner, the estimated 18TB a month of both sending and +receiving may cost between 0 and 10 Euro a month per node. + +## Discussion + +### Overview of Major Bandwidth Usage + +To determine which components of Tendermint were consuming the most bandwidth, +I gathered prometheus metrics from the [Blockpane][blockpane] validator running +on the Osmosis network for several hours. The data reveal that three message +types account for 98% of the total bandwidth consumed. These message types are +as follows: + + +1. [consensus.BlockPart][block-part-message] +2. [mempool.Txs][mempool-txs-message] +3. [consensus.Vote][vote-message] + + +The image below of p2p data collected from the Blockpane validator illustrate +the total bandwidth consumption of these three message types. + + +#### Send: + +##### Top 3 Percent: + +![](./images/top-3-percent-send.png) + +##### Rate For All Messages: + +![](./images/send-rate-all.png) + +#### Receive: + +##### Top 3 Percent: + +![](./images/top-3-percent-receive.png) + +##### Rate For All Messages: + +![](./images/receive-rate-all.png) + +### Investigation of Message Usage + +This section discusses the usage of each of the three highest consumption messages. +#### BlockPart Transmission + +Sending `BlockPart` messages consumes the most bandwidth out of all p2p +messages types as observed in the Blockpane Osmosis validator. In the almost 3 +hour observation, the validator sent about 20 gigabytes of `BlockPart` +messages. + +A block is proposed each round of Tendermint consensus. The paper does not +define a specific way that the block is to be transmitted, just that all +participants will receive it via a gossip network. + +The Go implementation of Tendermint transmits the block in 'parts'. It +serializes the block to wire-format proto and splits this byte representation +into a set of 4 kilobyte arrays and sends these arrays to its peers, each in a +separate message. + +The logic for sending `BlockPart` messages resides in the code for the +[consensus.Reactor][gossip-data-routine]. The consensus reactor starts a new +`gossipDataRoutine` for each peer it connects to. This routine repeatedly picks +a part of the block that Tendermint believes the peer does not know about yet +and gossips it to the peer. The set of `BlockParts` that Tendermint considers +its peer as having is only updated in one of four ways: + + + 1. Our peer tells us they have entered a new round [via a `NewRoundStep` +message][new-round-step-message-send]. This message is only sent when a node +moves to a new round or height and only resets the data we collect about a +peer's blockpart state. + 1. [We receive a block part from the peer][block-part-receive]. + 1. [We send][block-part-send-1] [the peer a block part][block-part-send-2]. + 1. Our peer tells us about the parts they have block [via `NewValidBlock` +messages][new-valid-block-message-send]. This message is only sent when the +peer has a quorum of prevotes or precommits for a block. + +Each node receives block parts from all of its peers. The particular block part +to send at any given time is randomly selected from the set of parts that the +peer node is not yet known to have. Given that these are the only times that +Tendermint learns of its peers' block parts, it's very likely that a node has +an incomplete understanding of its peers' block parts and is transmitting block +parts to a peer that the peer has received from some other node. + +Multiple potential mechanisms exist to reduce the number of duplicate block +parts a node receives. One set of mechanisms relies on more frequently +communicating the set of block parts a node needs to its peers. Another +potential mechanism requires a larger overhaul to the way blocks are gossiped +in the network. + +#### Mempool Tx Transmission + +The Tendermint mempool stages transactions that are yet to be committed to the +blockchain and communicates these transactions to its peers. Each message +contains one transaction. Data collected from the Blockpane node running on +Osmosis indicates that the validator sent about 12 gigabytes of `Txs` messages +during the nearly 3 hour observation period. + +The Tendermint mempool starts a new [broadcastTxRoutine][broadcast-tx-routine] +for each peer that it is informed of. The routine sends all transactions that +the mempool is aware of to all peers with few exceptions. The only exception is +if the mempool received a transaction from a peer, then it marks it as such and +won't resend to that peer. Otherwise, it retains no information about which +transactions it already sent to a peer. In some cases it may therefore resend +transactions the peer already has. This can occur if the mempool removes a +transaction from the `CList` data structure used to store the list of +transactions while it is about to be sent and if the transaction was the tail +of the `CList` during removal. This will be more likely to occur if a large +number of transactions from the end of the list are removed during `RecheckTx`, +since multiple transactions will become the tail and then be deleted. It is +unclear at the moment how frequently this occurs on production chains. + +Beyond ensuring that transactions are rebroadcast to peers less frequently, +there is not a simple scheme to communicate fewer transactions to peers. Peers +cannot communicate what transactions they need since they do not know which +transactions exist on the network. + +#### Vote Transmission + +Tendermint votes, both prevotes and precommits, are central to Tendermint +consensus and are gossiped by all nodes to all peers during each consensus +round. Data collected from the Blockpane node running on Osmosis indicates that +about 9 gigabytes of `Vote` messages were sent during the nearly 3 hour period +of observation. Examination of the [Vote message][vote-msg] indicates that it +contains 184 bytes of data, with the proto encoding adding a few additional +bytes when transmitting. + +The Tendermint consensus reactor starts a new +[gossipVotesRoutine][gossip-votes-routine] for each peer that it connects to. +The reactor sends all votes to all peers unless it knows that the peer already +has the vote or the reactor learns that the peer is in a different round and +that thus the vote no longer applies. Tendermint learns that a peer has a vote +in one of 4 ways: + + 1. Tendermint sent the peer the vote. + 1. Tendermint received the vote from the peer. + 1. The peer [sent a `HasVote` message][apply-has-vote]. This message is broadcast +to all peers [each time validator receives a vote it hasn't seen before +corresponding to its current height and round][publish-event-vote]. + 1. The peer [sent a `VoteSetBits` message][apply-vote-set-bits]. This message is +[sent as a response to a peer that sends a `VoteSetMaj23`][vote-set-bits-send]. + +Given that Tendermint informs all peers of _each_ vote message it receives, all +nodes should be well informed of which votes their peers have. Given that the +vote messages were the third largest consumer of bandwidth in the observation +on Osmosis, it's possible that this system is not currently working correctly. +Further analysis should examine where votes may be being retransmitted. + +### Suggested Improvements to Lower Message Transmission Bandwidth + +#### Gossip Known BlockPart Data + +The `BlockPart` messages, by far, account for the majority of the data sent to +each peer. At the moment, peers do not inform the node of which block parts +they already have. This means that each block part is _very likely_ to be +transmitted many times to each node. This frivolous consumption is even worse +in networks with large blocks. + +The very simple solution to this issue is to copy the technique used in +consensus for informing peers when the node receives a vote. The consensus +reactor can be augmented with a `HasBlockPart` message that is broadcast to +each peer every time the node receives a block part. By informing each peer +every time the node receives a block part, we can drastically reduce the amount +of duplicate data sent to each node. There would be no algorithmic way of +enforcing that a peer accurately reports its block parts, so providing this +message would be a somewhat altruistic action on the part of the node. Such a +system [has been proposed in the past][i627] as well, so this is certainly not +totally new ground. + +Measuring the size of duplicately received blockparts before and after this +change would help validate this approach. + +#### Compress Transmitted Data + +Tendermint's data is sent uncompressed on the wire. The messages are not +compressed before sending and the transport performs no compression either. +Some of the information communicated by Tendermint is a poor candidate for +compression: Data such as digital signatures and hashes have high entropy and +therefore do not compress well. However, transactions may contain lots of +information that has less entropy. Compression within Tendermint may be added +at several levels. Compression may be performed at the [Tendermint 'packet' +level][must-wrap-packet] or at the [Tendermint message send +level][message-send]. + +#### Transmit Less Data During Block Gossip + +Block, vote, and mempool gossiping transmit much of same data. The mempool +reactor gossips candidate transactions to each peer. The consensus reactor, +when gossiping the votes, sends vote metadata and the digital signature of that +signs over that metadata. Finally, when a block is proposed, the proposing node +amalgamates the received votes, a set of transaction, and adds a header to +produce the block. This block is then serialized and gossiped as a list of +bytes. However, the data that the block contains, namely the votes and the +transactions were most likely _already transmitted to the nodes on the network_ +via mempool transaction gossip and consensus vote gossip. + +Therefore, block gossip can be updated to transmit a representation of the data +contained in the block that assumes the peers will already have most of this +data. Namely, the block gossip can be updated to only send 1) a list of +transaction hashes and 2) a bit array of votes selected for the block along +with the header and other required block metadata. + +This new proposed method for gossiping block data could accompany a slight +update to the mempool transaction gossip and consensus vote gossip. Since all +of the contents of each block will not be gossiped together, it's possible that +some nodes are missing a proposed transaction or the vote of a validator +indicated in the new block gossip format during block gossip. The mempool and +consensus reactors may therefore be updated to provide a `NeedTxs` and +`NeedVotes` message. Each of these messages would allow a node to request a set +of data from their peers. When a node receives one of these, it will then +transmit the Tx/Votes indicate in the associated message regardless of whether +it believes it has transmitted them to the peer before. The gossip layer will +ensure that each peer eventually receives all of the data in the block. +However, if a transaction is needed immediately by a peer so that it can verify +and execute a block during consensus, a mechanism such as the `NeedTxs` and +`NeedVotes` messages should be added to ensure it receives the messages +quickly. + +The same logic may applied for evidence transmission as well, since all nodes +should receive evidence and therefore do not need to re-transmit it in a block +part. + +A similar idea has been proposed in the past as [Compact Block +Propagation][compact-block-propagation]. + + +## References + +[blockpane]: https://www.mintscan.io/osmosis/validators/osmovaloper1z0sh4s80u99l6y9d3vfy582p8jejeeu6tcucs2 +[block-part-message]: https://github.com/tendermint/tendermint/blob/ff0f98892f24aac11e46aeff2b6d2c0ad816701a/proto/tendermint/consensus/types.proto#L44 +[mempool-txs-message]: https://github.com/tendermint/tendermint/blob/ff0f98892f24aac11e46aeff2b6d2c0ad816701a/proto/tendermint/mempool/types.proto#L6 +[vote-message]: https://github.com/tendermint/tendermint/blob/ff0f98892f24aac11e46aeff2b6d2c0ad816701a/proto/tendermint/consensus/types.proto#L51 +[gossip-data-routine]: https://github.com/tendermint/tendermint/blob/ff0f98892f24aac11e46aeff2b6d2c0ad816701a/consensus/reactor.go#L537 +[block-part-receive]: https://github.com/tendermint/tendermint/blob/ff0f98892f24aac11e46aeff2b6d2c0ad816701a/consensus/reactor.go#L324 +[block-part-send-1]: https://github.com/tendermint/tendermint/blob/ff0f98892f24aac11e46aeff2b6d2c0ad816701a/consensus/reactor.go#L566 +[block-part-send-2]: https://github.com/tendermint/tendermint/blob/ff0f98892f24aac11e46aeff2b6d2c0ad816701a/consensus/reactor.go#L684. +[new-valid-block-message-send]: https://github.com/tendermint/tendermint/blob/ff0f98892f24aac11e46aeff2b6d2c0ad816701a/consensus/reactor.go#L268 +[new-round-step-message-send]: https://github.com/tendermint/tendermint/blob/ff0f98892f24aac11e46aeff2b6d2c0ad816701a/consensus/reactor.go#L266 +[broadcast-tx-routine]: https://github.com/tendermint/tendermint/blob/ff0f98892f24aac11e46aeff2b6d2c0ad816701a/mempool/v0/reactor.go#L197 +[gossip-votes-routine]: https://github.com/tendermint/tendermint/blob/ff0f98892f24aac11e46aeff2b6d2c0ad816701a/consensus/reactor.go#L694 +[apply-has-vote]: https://github.com/tendermint/tendermint/blob/ff0f98892f24aac11e46aeff2b6d2c0ad816701a/consensus/reactor.go#L1429 +[apply-vote-set-bits]: https://github.com/tendermint/tendermint/blob/ff0f98892f24aac11e46aeff2b6d2c0ad816701a/consensus/reactor.go#L1445 +[publish-event-vote]: https://github.com/tendermint/tendermint/blob/ff0f98892f24aac11e46aeff2b6d2c0ad816701a/consensus/state.go#L2083 +[vote-set-bits-send]: https://github.com/tendermint/tendermint/blob/ff0f98892f24aac11e46aeff2b6d2c0ad816701a/consensus/reactor.go#L306 +[must-wrap-packet]: https://github.com/tendermint/tendermint/blob/ff0f98892f24aac11e46aeff2b6d2c0ad816701a/p2p/conn/connection.go#L889-L918 +[message-send]: https://github.com/tendermint/tendermint/blob/ff0f98892f24aac11e46aeff2b6d2c0ad816701a/p2p/peer.go#L285 +[gcloud-pricing]: https://cloud.google.com/vpc/network-pricing#vpc-pricing +[hetzner-pricing]: https://docs.hetzner.com/robot/general/traffic +[vote-msg]: https://github.com/tendermint/tendermint/blob/ff0f98892f24aac11e46aeff2b6d2c0ad816701a/proto/tendermint/types/types.pb.go#L468 +[i627]: https://github.com/tendermint/tendermint/issues/627 +[compact-block-propagation]: https://github.com/tendermint/tendermint/issues/7932 diff --git a/cometbft/v0.38/docs/tools/Overview.mdx b/cometbft/v0.38/docs/tools/Overview.mdx new file mode 100644 index 00000000..a3ec29a3 --- /dev/null +++ b/cometbft/v0.38/docs/tools/Overview.mdx @@ -0,0 +1,20 @@ +--- +order: 1 +parent: + title: Tools + order: 5 +--- + +# Overview + +CometBFT has some tools that are associated with it for: + +- [Debugging](./debugging.md) +- [Benchmarking](#benchmarking) + +## Benchmarking + +- [https://github.com/informalsystems/tm-load-test](https://github.com/informalsystems/tm-load-test) + +`tm-load-test` is a distributed load testing tool (and framework) for load +testing CometBFT networks. diff --git a/cometbft/v0.38/docs/tools/debugging.mdx b/cometbft/v0.38/docs/tools/debugging.mdx new file mode 100644 index 00000000..4f7af58b --- /dev/null +++ b/cometbft/v0.38/docs/tools/debugging.mdx @@ -0,0 +1,105 @@ +--- +order: 1 +--- + +# Debugging + +## CometBFT debug kill + +CometBFT comes with a `debug` sub-command that allows you to kill a live +CometBFT process while collecting useful information in a compressed archive. +The information includes the configuration used, consensus state, network +state, the node' status, the WAL, and even the stack trace of the process +before exit. These files can be useful to examine when debugging a faulty +CometBFT process. + +```bash +cometbft debug kill --home= +``` + +will write debug info into a compressed archive. The archive will contain the +following: + +```sh +├── config.toml +├── consensus_state.json +├── net_info.json +├── stacktrace.out +├── status.json +└── wal +``` + +Under the hood, `debug kill` fetches info from `/status`, `/net_info`, and +`/dump_consensus_state` HTTP endpoints, and kills the process with `-6`, which +catches the go-routine dump. + +## CometBFT debug dump + +Also, the `debug dump` sub-command allows you to dump debugging data into +compressed archives at a regular interval. These archives contain the goroutine +and heap profiles in addition to the consensus state, network info, node +status, and even the WAL. + +```bash +cometbft debug dump --home= +``` + +will perform similarly to `kill` except it only polls the node and +dumps debugging data every frequency seconds to a compressed archive under a +given destination directory. Each archive will contain: + +```sh +├── consensus_state.json +├── goroutine.out +├── heap.out +├── net_info.json +├── status.json +└── wal +``` + +Note: goroutine.out and heap.out will only be written if a profile address is +provided and is operational. This command is blocking and will log any error. + +## CometBFT Inspect + +CometBFT includes an `inspect` command for querying CometBFT's state store and block +store over CometBFT RPC. + +When the CometBFT consensus engine detects inconsistent state, it will crash the +entire CometBFT process. +While in this inconsistent state, a node running CometBFT will not start up. +The `inspect` command runs only a subset of CometBFT's RPC endpoints for querying the block store +and state store. +`inspect` allows operators to query a read-only view of the stage. +`inspect` does not run the consensus engine at all and can therefore be used to debug +processes that have crashed due to inconsistent state. + +### Running inspect + +Start up the `inspect` tool on the machine where CometBFT crashed using: +```bash +cometbft inspect --home= +``` + +`inspect` will use the data directory specified in your CometBFT configuration file. +`inspect` will also run the RPC server at the address specified in your CometBFT configuration file. + +### Using inspect + +With the `inspect` server running, you can access RPC endpoints that are critically important +for debugging. +Calling the `/status`, `/consensus_state` and `/dump_consensus_state` RPC endpoint +will return useful information about the CometBFT consensus state. + +To start the `inspect` process, run +```bash +cometbft inspect +``` + +### RPC endpoints + +The list of available RPC endpoints can be found by making a request to the RPC port. +For an `inspect` process running on `127.0.0.1:26657`, navigate your browser to +`http://127.0.0.1:26657/` to retrieve the list of enabled RPC endpoints. + +Additional information on the CometBFT RPC endpoints can be found in the [rpc documentation](https://docs.cometbft.com/v0.38/rpc). diff --git a/cometbft/v0.38/spec/CometBFT-Spec.mdx b/cometbft/v0.38/spec/CometBFT-Spec.mdx new file mode 100644 index 00000000..8b9a1c7a --- /dev/null +++ b/cometbft/v0.38/spec/CometBFT-Spec.mdx @@ -0,0 +1,94 @@ +--- +order: 1 +title: Overview +parent: + title: Spec + order: 7 +--- +{/* trigger rebuild */} + +# CometBFT Spec + +This is a markdown specification of CometBFT. +It defines the base data structures, how they are validated, +and how they are communicated over the network. + +If you find discrepancies between the spec and the code that +do not have an associated issue or pull request on github, +please submit them to our [bug bounty](https://github.com/cometbft/cometbft#security)! + +## Contents + +- [Overview](#overview) + +### Data Structures + +- [Encoding and Digests](/cometbft/v0.38/spec/core/encoding) +- [Blockchain](/cometbft/v0.38/spec/core/Data_structures) +- [State](/cometbft/v0.38/spec/core/state) + +### Consensus Protocol + +- [Consensus Algorithm](/cometbft/v0.38/spec/consensus/Byzantine-Consensus-Algorithm) +- [Creating a proposal](/cometbft/v0.38/spec/consensus/Creating-Proposal) +- [Time](/cometbft/v0.38/spec/consensus/BFT-Time) +- [Light-Client](/cometbft/v0.38/spec/consensus/Light-Client) + +### P2P and Network Protocols + +- [The Base P2P Layer](/cometbft/v0.38/spec/p2p/legacy-docs/Peer-Discovery): multiplex the protocols ("reactors") on authenticated and encrypted TCP connections +- [Peer Exchange (PEX)](/cometbft/v0.38/spec/p2p/legacy-docs/messages/Peer-Exchange): gossip known peer addresses so peers can find each other +- [Block Sync](/cometbft/v0.38/spec/p2p/legacy-docs/messages/block-sync): gossip blocks so peers can catch up quickly +- [Consensus](/cometbft/v0.38/spec/p2p/legacy-docs/messages/consensus): gossip votes and block parts so new blocks can be committed +- [Mempool](/cometbft/v0.38/spec/p2p/legacy-docs/messages/mempool): gossip transactions so they get included in blocks +- [Evidence](/cometbft/v0.38/spec/p2p/legacy-docs/messages/evidence): sending invalid evidence will stop the peer + +### RPC + +- [RPC SPEC](/cometbft/v0.38/spec/rpc/Rpc-Spe): Specification of the CometBFT remote procedure call interface. + +### Software + +- [ABCI](/cometbft/v0.38/spec/abci/Overview): Details about interactions between the + application and consensus engine over ABCI +- [Write-Ahead Log](/cometbft/v0.38/spec/consensus/WAL): Details about how the consensus + engine preserves data and recovers from crash failures + +## Overview + +CometBFT provides Byzantine Fault Tolerant State Machine Replication using +hash-linked batches of transactions. Such transaction batches are called "blocks". +Hence, CometBFT defines a "blockchain". + +Each block in CometBFT has a unique index - its Height. +Heights in the blockchain are monotonic. +Each block is committed by a known set of weighted Validators. +Membership and weighting within this validator set may change over time. +CometBFT guarantees the safety and liveness of the blockchain +as long as less than 1/3 of the total weight of the Validator set +is malicious or faulty. + +A commit in CometBFT is a set of signed messages from more than 2/3 of +the total weight of the current Validator set. Validators take turns proposing +blocks and voting on them. Once enough votes are received, the block is considered +committed. These votes are included in the _next_ block as proof that the previous block +was committed - they cannot be included in the current block, as that block has already been +created. + +Once a block is committed, it can be executed against an application. +The application returns results for each of the transactions in the block. +The application can also return changes to be made to the validator set, +as well as a cryptographic digest of its latest state. + +CometBFT is designed to enable efficient verification and authentication +of the latest state of the blockchain. To achieve this, it embeds +cryptographic commitments to certain information in the block "header". +This information includes the contents of the block (eg. the transactions), +the validator set committing the block, as well as the various results returned by the application. +Note, however, that block execution only occurs _after_ a block is committed. +Thus, application results can only be included in the _next_ block. + +Also note that information like the transaction results and the validator set are never +directly included in the block - only their cryptographic digests (Merkle roots) are. +Hence, verification of a block requires a separate data structure to store this information. +We call this the `State`. Block verification also requires access to the previous block. diff --git a/cometbft/v0.38/spec/abci/Client-and-server.mdx b/cometbft/v0.38/spec/abci/Client-and-server.mdx new file mode 100644 index 00000000..fb30e862 --- /dev/null +++ b/cometbft/v0.38/spec/abci/Client-and-server.mdx @@ -0,0 +1,94 @@ +--- +order: 5 +title: Client and Server +--- + +# Client and Server + +This section is for those looking to implement their own ABCI Server, perhaps in +a new programming language. + +You are expected to have read all previous sections of ABCI++ specification, namely +[Basic Concepts](./abci%2B%2B_basic_concepts.md), +[Methods](./abci%2B%2B_methods.md), +[Application Requirements](./abci%2B%2B_app_requirements.md), and +[Expected Behavior](./abci%2B%2B_comet_expected_behavior.md). + +## Message Protocol and Synchrony + +The message protocol consists of pairs of requests and responses defined in the +[protobuf file](https://github.com/cometbft/cometbft/blob/v0.38.x/proto/tendermint/abci/types.proto). + +Some messages have no fields, while others may include byte-arrays, strings, integers, +or custom protobuf types. + +For more details on protobuf, see the [documentation](https://developers.google.com/protocol-buffers/docs/overview). + +{/* +As of v0.36 requests are synchronous. For each of ABCI++'s four connections (see +[Connections](./abci%2B%2B_app_requirements.md)), when CometBFT issues a request to the +Application, it will wait for the response before continuing execution. As a side effect, +requests and responses are ordered for each connection, but not necessarily across connections. +*/} +## Server Implementations + +To use ABCI in your programming language of choice, there must be an ABCI +server in that language. CometBFT supports four implementations of the ABCI server: + +- in CometBFT's repository: + - In-process + - ABCI-socket + - GRPC +- [tendermint-rs](https://github.com/informalsystems/tendermint-rs) +- [tower-abci](https://github.com/penumbra-zone/tower-abci) + +The implementations in CometBFT's repository can be tested using `abci-cli` by setting +the `--abci` flag appropriately. + +See examples, in various stages of maintenance, in +[Go](https://github.com/cometbft/cometbft/tree/master/abci/server), +[JavaScript](https://github.com/tendermint/js-abci), +[C++](https://github.com/mdyring/cpp-tmsp), and +[Java](https://github.com/jTendermint/jabci). + +### In Process + +The simplest implementation uses function calls in Golang. +This means ABCI applications written in Golang can be linked with CometBFT and run as a single binary. + +### GRPC + +If you are not using Golang, +but [GRPC](https://grpc.io/) is available in your language, this is the easiest approach, +though it will have significant performance overhead. + +Please check GRPC's documentation to know to set up the Application as an +ABCI GRPC server. + +### Socket + +The CometBFT Socket Protocol is an asynchronous, raw socket server protocol which provides ordered +message passing over Unix or TCP sockets. Messages are serialized using Protobuf3 and length-prefixed +with an [unsigned varint](https://developers.google.com/protocol-buffers/docs/encoding?csw=1#varints) + +If gRPC is not available in your language, or you require higher performance, or +otherwise enjoy programming, you may implement your own ABCI server using the +CometBFT Socket Protocol. The first step is still to auto-generate the +relevant data types and codec in your language using `protoc`, and then you need to +ensure you handle the unsigned `varint`-based message length encoding scheme +when reading and writing messages to the socket. + +Note that our length prefixing scheme does not apply to gRPC. + +Also note that your ABCI server must be able to handle multiple connections, +as CometBFT uses four connections. + +## Client + +There are currently two use-cases for an ABCI client. One is testing +tools that allow ABCI requests to be sent to the actual application via +command line. An example of this is `abci-cli`, which accepts CLI commands +to send corresponding ABCI requests. +The other is a consensus engine, such as CometBFT, +which makes ABCI requests to the application as prescribed by the consensus +algorithm used. diff --git a/cometbft/v0.38/spec/abci/CometBFTs-expected-behavior.mdx b/cometbft/v0.38/spec/abci/CometBFTs-expected-behavior.mdx new file mode 100644 index 00000000..092d49e0 --- /dev/null +++ b/cometbft/v0.38/spec/abci/CometBFTs-expected-behavior.mdx @@ -0,0 +1,282 @@ +--- +order: 4 +title: CometBFT's expected behavior +--- + +# CometBFT's expected behavior + +## Valid method call sequences + +This section describes what the Application can expect from CometBFT. + +The Tendermint consensus algorithm, currently adopted in CometBFT, is designed to protect safety under any network conditions, as long as +less than 1/3 of validators' voting power is byzantine. Most of the time, though, the network will behave +synchronously, no process will fall behind, and there will be no byzantine process. The following describes +what will happen during a block height _h_ in these frequent, benign conditions: + +* Consensus will decide in round 0, for height _h_; +* `PrepareProposal` will be called exactly once at the proposer process of round 0, height _h_; +* `ProcessProposal` will be called exactly once at all processes, and + will return _accept_ in its `Response*`; +* `ExtendVote` will be called exactly once at all processes; +* `VerifyVoteExtension` will be called exactly _n-1_ times at each validator process, where _n_ is + the number of validators, and will always return _accept_ in its `Response*`; +* `FinalizeBlock` will be called exactly once at all processes, conveying the same prepared + block that all calls to `PrepareProposal` and `ProcessProposal` had previously reported for + height _h_; and +* `Commit` will finally be called exactly once at all processes at the end of height _h_. + +However, the Application logic must be ready to cope with any possible run of the consensus algorithm for a given +height, including bad periods (byzantine proposers, network being asynchronous). +In these cases, the sequence of calls to ABCI++ methods may not be so straightforward, but +the Application should still be able to handle them, e.g., without crashing. +The purpose of this section is to define what these sequences look like in a precise way. + +As mentioned in the [Basic Concepts](./abci%2B%2B_basic_concepts.md) section, CometBFT +acts as a client of ABCI++ and the Application acts as a server. Thus, it is up to CometBFT to +determine when and in which order the different ABCI++ methods will be called. A well-written +Application design should consider _any_ of these possible sequences. + +The following grammar, written in case-sensitive Augmented Backus–Naur form (ABNF, specified +in [IETF rfc7405](https://datatracker.ietf.org/doc/html/rfc7405)), specifies all possible +sequences of calls to ABCI++, taken by a **correct process**, across all heights from the genesis block, +including recovery runs, from the point of view of the Application. + +```abnf +start = clean-start / recovery + +clean-start = init-chain [state-sync] consensus-exec +state-sync = *state-sync-attempt success-sync info +state-sync-attempt = offer-snapshot *apply-chunk +success-sync = offer-snapshot 1*apply-chunk + +recovery = info consensus-exec + +consensus-exec = (inf)consensus-height +consensus-height = *consensus-round decide commit +consensus-round = proposer / non-proposer + +proposer = *got-vote [prepare-proposal [process-proposal]] [extend] +extend = *got-vote extend-vote *got-vote +non-proposer = *got-vote [process-proposal] [extend] + +init-chain = %s"" +offer-snapshot = %s"" +apply-chunk = %s"" +info = %s"" +prepare-proposal = %s"" +process-proposal = %s"" +extend-vote = %s"" +got-vote = %s"" +decide = %s"" +commit = %s"" +``` + +We have kept some ABCI methods out of the grammar, in order to keep it as clear and concise as possible. +A common reason for keeping all these methods out is that they all can be called at any point in a sequence defined +by the grammar above. Other reasons depend on the method in question: + +* `Echo` and `Flush` are only used for debugging purposes. Further, their handling by the Application should be trivial. +* `CheckTx` is detached from the main method call sequence that drives block execution. +* `Query` provides read-only access to the current Application state, so handling it should also be independent from + block execution. +* Similarly, `ListSnapshots` and `LoadSnapshotChunk` provide read-only access to the Application's previously created + snapshots (if any), and help populate the parameters of `OfferSnapshot` and `ApplySnapshotChunk` at a process performing + state-sync while bootstrapping. Unlike `ListSnapshots` and `LoadSnapshotChunk`, both `OfferSnapshot` + and `ApplySnapshotChunk` _are_ included in the grammar. + +Finally, method `Info` is a special case. The method's purpose is three-fold, it can be used + +1. as part of handling an RPC call from an external client, +2. as a handshake between CometBFT and the Application upon recovery to check whether any blocks need + to be replayed, and +3. at the end of _state-sync_ to verify that the correct state has been reached. + +We have left `Info`'s first purpose out of the grammar for the same reasons as all the others: it can happen +at any time, and has nothing to do with the block execution sequence. The second and third purposes, on the other +hand, are present in the grammar. + +Let us now examine the grammar line by line, providing further details. + +* When a process starts, it may do so for the first time or after a crash (it is recovering). + +>```abnf +>start = clean-start / recovery +>``` + +* If the process is starting from scratch, CometBFT first calls `InitChain`, then it may optionally + start a _state-sync_ mechanism to catch up with other processes. Finally, it enters normal + consensus execution. + +>```abnf +>clean-start = init-chain [state-sync] consensus-exec +>``` + +* In _state-sync_ mode, CometBFT makes one or more attempts at synchronizing the Application's state. + At the beginning of each attempt, it offers the Application a snapshot found at another process. + If the Application accepts the snapshot, a sequence of calls to `ApplySnapshotChunk` method follow + to provide the Application with all the snapshots needed, in order to reconstruct the state locally. + A successful attempt must provide at least one chunk via `ApplySnapshotChunk`. + At the end of a successful attempt, CometBFT calls `Info` to make sure the reconstructed state's + _AppHash_ matches the one in the block header at the corresponding height. Note that the state + of the application does not contain vote extensions itself. The application can rely on + [CometBFT to ensure](https://github.com/cometbft/cometbft/blob/v0.38.x/docs/rfc/rfc-100-abci-vote-extension-propag.md#base-implementation-persist-and-propagate-extended-commit-history) + the node has all the relevant data to proceed with the execution beyond this point. + +>```abnf +>state-sync = *state-sync-attempt success-sync info +>state-sync-attempt = offer-snapshot *apply-chunk +>success-sync = offer-snapshot 1*apply-chunk +>``` + +* In recovery mode, CometBFT first calls `Info` to know from which height it needs to replay decisions + to the Application. After this, CometBFT enters consensus execution, first in replay mode and then + in normal mode. + +>```abnf +>recovery = info consensus-exec +>``` + +* The non-terminal `consensus-exec` is a key point in this grammar. It is an infinite sequence of + consensus heights. The grammar is thus an + [omega-grammar](https://dl.acm.org/doi/10.5555/2361476.2361481), since it produces infinite + sequences of terminals (i.e., the API calls). + +>```abnf +>consensus-exec = (inf)consensus-height +>``` + +* A consensus height consists of zero or more rounds before deciding and executing via a call to + `FinalizeBlock`, followed by a call to `Commit`. In each round, the sequence of method calls + depends on whether the local process is the proposer or not. Note that, if a height contains zero + rounds, this means the process is replaying an already decided value (catch-up mode). + When calling `FinalizeBlock` with a block, the consensus algorithm run by CometBFT guarantees + that at least one non-byzantine validator has run `ProcessProposal` on that block. + + +>```abnf +>consensus-height = *consensus-round decide commit +>consensus-round = proposer / non-proposer +>``` + +* For every round, if the local process is the proposer of the current round, CometBFT calls `PrepareProposal`. + A successful execution of `PrepareProposal` results in a proposal block being (i) signed and (ii) stored + (e.g., in stable storage). + + A crash during this step will direct how the node proceeds the next time it is executed, for the same round, after restarted. + If it crashed before (i), then, during the recovery, `PrepareProposal` will execute as if for the first time. + Following a crash between (i) and (ii) and in (the likely) case `PrepareProposal` produces a different block, + the signing of this block will fail, which means that the new block will not be stored or broadcast. + If the crash happened after (ii), then signing fails but nothing happens to the stored block. + + If a block was stored, it is sent to all validators, including the proposer. + Receiving a proposal block triggers `ProcessProposal` with such a block. + + Then, optionally, the Application is + asked to extend its vote for that round. Calls to `VerifyVoteExtension` can come at any time: the + local process may be slightly late in the current round, or votes may come from a future round + of this height. + +>```abnf +>proposer = *got-vote [prepare-proposal [process-proposal]] [extend] +>extend = *got-vote extend-vote *got-vote +>``` + +* Also for every round, if the local process is _not_ the proposer of the current round, CometBFT + will call `ProcessProposal` at most once. + Under certain conditions, CometBFT may not call `ProcessProposal` in a round; + see [this section](./abci++_example_scenarios.md#scenario-3) for an example. + At most one call to `ExtendVote` may occur only after + `ProcessProposal` is called. A number of calls to `VerifyVoteExtension` can occur in any order + with respect to `ProcessProposal` and `ExtendVote` throughout the round. The reasons are the same + as above, namely, the process running slightly late in the current round, or votes from future + rounds of this height received. + +>```abnf +>non-proposer = *got-vote [process-proposal] [extend] +>``` + +* Finally, the grammar describes all its terminal symbols, which denote the different ABCI++ method calls that + may appear in a sequence. + +>```abnf +>init-chain = %s"" +>offer-snapshot = %s"" +>apply-chunk = %s"" +>info = %s"" +>prepare-proposal = %s"" +>process-proposal = %s"" +>extend-vote = %s"" +>got-vote = %s"" +>decide = %s"" +>commit = %s"" +>``` + +## Adapting existing Applications that use ABCI + +In some cases, an existing Application using the legacy ABCI may need to be adapted to work with ABCI++ +with as minimal changes as possible. In this case, of course, ABCI++ will not provide any advantage with respect +to the existing implementation, but will keep the same guarantees already provided by ABCI. +Here is how ABCI++ methods should be implemented. + +First of all, all the methods that did not change from ABCI 0.17.0 to ABCI 2.0, namely `Echo`, `Flush`, `Info`, `InitChain`, +`Query`, `CheckTx`, `ListSnapshots`, `LoadSnapshotChunk`, `OfferSnapshot`, and `ApplySnapshotChunk`, do not need +to undergo any changes in their implementation. + +As for the new methods: + +* `PrepareProposal` must create a list of [transactions](./abci++_methods.md#prepareproposal) + by copying over the transaction list passed in `RequestPrepareProposal.txs`, in the same order. + + The Application must check whether the size of all transactions exceeds the byte limit + (`RequestPrepareProposal.max_tx_bytes`). If so, the Application must remove transactions at the + end of the list until the total byte size is at or below the limit. +* `ProcessProposal` must set `ResponseProcessProposal.status` to _accept_ and return. +* `ExtendVote` is to set `ResponseExtendVote.extension` to an empty byte array and return. +* `VerifyVoteExtension` must set `ResponseVerifyVoteExtension.accept` to _true_ if the extension is + an empty byte array and _false_ otherwise, then return. +* `FinalizeBlock` is to coalesce the implementation of methods `BeginBlock`, `DeliverTx`, and + `EndBlock`. Legacy applications looking to reuse old code that implemented `DeliverTx` should + wrap the legacy `DeliverTx` logic in a loop that executes one transaction iteration per + transaction in `RequestFinalizeBlock.tx`. + +Finally, `Commit`, which is kept in ABCI++, no longer returns the `AppHash`. It is now up to +`FinalizeBlock` to do so. Thus, a slight refactoring of the old `Commit` implementation will be +needed to move the return of `AppHash` to `FinalizeBlock`. + +## Accommodating for vote extensions + +In a manner transparent to the application, CometBFT ensures the node is provided with all +the data it needs to participate in consensus. + +In the case of recovering from a crash, or joining the network via state sync, CometBFT will make +sure the node acquires the necessary vote extensions before switching to consensus. + +If a node is already in consensus but falls behind, during catch-up, CometBFT will provide the node with +vote extensions from past heights by retrieving the extensions within `ExtendedCommit` for old heights that it had previously stored. + +We realize this is sub-optimal due to the increase in storage needed to store the extensions, we are +working on an optimization of this implementation which should alleviate this concern. +However, the application can use the existing `retain_height` parameter to decide how much +history it wants to keep, just as is done with the block history. The network-wide implications +of the usage of `retain_height` stay the same. +The decision to store +historical commits and potential optimizations, are discussed in detail in [RFC-100](https://github.com/cometbft/cometbft/blob/v0.38.x/docs/rfc/rfc-100-abci-vote-extension-propag.md#current-limitations-and-possible-implementations) + +## Handling upgrades to ABCI 2.0 + +If applications upgrade to ABCI 2.0, CometBFT internally ensures that the [application setup](./abci%2B%2B_app_requirements.md#application-configuration-required-to-switch-to-abci-20) is reflected in its operation. +CometBFT retrieves from the application configuration the value of `VoteExtensionsEnableHeight`( _he_,), +the height at which vote extensions are required for consensus to proceed, and uses it to determine the data it stores and data it sends to a peer that is catching up. + +Namely, upon saving the block for a given height _h_ in the block store at decision time + +* if _h ≥ he_, the corresponding extended commit that was used to decide locally is saved as well +* if _h < he_, there are no changes to the data saved + +In the catch-up mechanism, when a node _f_ realizes that another peer is at height _hp_, which is more than 2 heights behind height _hf_, + +* if _hp ≥ he_, _f_ uses the extended commit to + reconstruct the precommit votes with their corresponding extensions +* if _hp < he_, _f_ uses the canonical commit to reconstruct the precommit votes, + as done for ABCI 1.0 and earlier. diff --git a/cometbft/v0.38/spec/abci/Introduction.mdx b/cometbft/v0.38/spec/abci/Introduction.mdx new file mode 100644 index 00000000..b7aad3c0 --- /dev/null +++ b/cometbft/v0.38/spec/abci/Introduction.mdx @@ -0,0 +1,166 @@ +--- +order: 6 +title: ABCI++ extra +--- +{/* trigger rebuild */} + +# Introduction + +In the section [CometBFT's expected behaviour](./abci++_comet_expected_behavior.md#valid-method-call-sequences), +we presented the most common behaviour, usually referred to as the good case. +However, the grammar specified in the same section is more general and covers more scenarios +that an Application designer needs to account for. + +In this section, we give more information about these possible scenarios. We focus on methods +introduced by ABCI++: `PrepareProposal` and `ProcessProposal`. Specifically, we concentrate +on the part of the grammar presented below. + +```abnf +consensus-height = *consensus-round decide commit +consensus-round = proposer / non-proposer + +proposer = [prepare-proposal process-proposal] +non-proposer = [process-proposal] +``` + +We can see from the grammar that we can have several rounds before deciding a block. The reasons +why one round may not be enough are: + +* network asynchrony, and +* a Byzantine process being the proposer. + +If we assume that the consensus algorithm decides on block $X$ in round $r$, in the rounds +$r' <= r$, CometBFT can exhibit any of the following behaviours: + +1. Call `PrepareProposal` and/or `ProcessProposal` for block $X$. +1. Call `PrepareProposal` and/or `ProcessProposal` for block $Y \neq X$. +1. Does not call `PrepareProposal` and/or `ProcessProposal`. + +In the rounds in which the process is the proposer, CometBFT's `PrepareProposal` call is always followed by the +`ProcessProposal` call. The reason is that the process also broadcasts the proposal to itself, which is locally delivered and triggers the `ProcessProposal` call. +The proposal processed by `ProcessProposal` is the same as what was returned by any of the preceding `PrepareProposal` invoked for the same height and round. +While in the absence of restarts there is only one such preceding invocations, if the proposer restarts there could have been one extra invocation to `PrepareProposal` for each restart. + +As the number of rounds the consensus algorithm needs to decide in a given run is a priori unknown, the +application needs to account for any number of rounds, where each round can exhibit any of these three +behaviours. Recall that the application is unaware of the internals of consensus and thus of the rounds. + +# Possible scenarios + +The unknown number of rounds we can have when following the consensus algorithm yields a vast number of +scenarios we can expect. Listing them all is unfeasible. However, here we give several of them and draw the +main conclusions. Specifically, we will show that before block $X$ is decided: + +1. On a correct node, `PrepareProposal` may be called multiple times and for different blocks ([**Scenario 1**](#scenario-1)). +1. On a correct node, `ProcessProposal` may be called multiple times and for different blocks ([**Scenario 2**](#scenario-2)). +1. On a correct node, `PrepareProposal` and `ProcessProposal` for block $X$ may not be called ([**Scenario 3**](#scenario-3)). +1. On a correct node, `PrepareProposal` and `ProcessProposal` may not be called at all ([**Scenario 4**](#scenario-4)). + + +## Basic information + +Each scenario is presented from the perspective of a process $p$. More precisely, we show what happens in +each round's $step$ of the [Tendermint consensus algorithm](https://arxiv.org/pdf/1807.04938.pdf). While in +practice the consensus algorithm works with respect to voting power of the validators, in this document +we refer to number of processes (e.g., $n$, $f+1$, $2f+1$) for simplicity. The legend is below: + +### Round X + +1. **Propose:** Describes what happens while $step_p = propose$. +1. **Prevote:** Describes what happens while $step_p = prevote$. +1. **Precommit:** Describes what happens while $step_p = precommit$. + +## Scenario 1 + +$p$ calls `ProcessProposal` many times with different values. + +### Round 0 + +1. **Propose:** The proposer of this round is a Byzantine process, and it chooses not to send the proposal +message. Therefore, $p$'s $timeoutPropose$ expires, it sends $Prevote$ for $nil$, and it does not call +`ProcessProposal`. All correct processes do the same. +1. **Prevote:** $p$ eventually receives $2f+1$ $Prevote$ messages for $nil$ and starts $timeoutPrevote$. +When $timeoutPrevote$ expires it sends $Precommit$ for $nil$. +1. **Precommit:** $p$ eventually receives $2f+1$ $Precommit$ messages for $nil$ and starts $timeoutPrecommit$. +When it expires, it moves to the next round. + +### Round 1 + +1. **Propose:** A correct process is the proposer in this round. Its $validValue$ is $nil$, and it is free +to generate and propose a new block $Y$. Process $p$ receives this proposal in time, calls `ProcessProposal` +for block $Y$, and broadcasts a $Prevote$ message for it. +1. **Prevote:** Due to network asynchrony less than $2f+1$ processes send $Prevote$ for this block. +Therefore, $p$ does not update $validValue$ in this round. +1. **Precommit:** Since less than $2f+1$ processes send $Prevote$, no correct process will lock on this +block and send $Precommit$ message. As a consequence, $p$ does not decide on $Y$. + +### Round 2 + +1. **Propose:** Same as in [**Round 1**](#round-1), just another correct process is the proposer, and it +proposes another value $Z$. Process $p$ receives the proposal on time, calls `ProcessProposal` for new block +$Z$, and broadcasts a $Prevote$ message for it. +1. **Prevote:** Same as in [**Round 1**](#round-1). +1. **Precommit:** Same as in [**Round 1**](#round-1). + + +Rounds like these can continue until we have a round in which process $p$ updates its $validValue$ or until +we reach round $r$ where process $p$ decides on a block. After that, it will not call `ProcessProposal` +anymore for this height. + +## Scenario 2 + +$p$ calls `PrepareProposal` many times with different values. + +### Round 0 + +1. **Propose:** Process $p$ is the proposer in this round. Its $validValue$ is $nil$, and it is free to +generate and propose new block $Y$. Before proposing, it calls `PrepareProposal` for $Y$. After that, it +broadcasts the proposal, delivers it to itself, calls `ProcessProposal` and broadcasts $Prevote$ for it. +1. **Prevote:** Due to network asynchrony less than $2f+1$ processes receive the proposal on time and send +$Prevote$ for it. Therefore, $p$ does not update $validValue$ in this round. +1. **Precommit:** Since less than $2f+1$ processes send $Prevote$, no correct process will lock on this +block and send non-$nil$ $Precommit$ message. As a consequence, $p$ does not decide on $Y$. + +After this round, we can have multiple rounds like those in [Scenario 1](#scenario-1). The important thing +is that process $p$ should not update its $validValue$. Consequently, when process $p$ reaches the round +when it is again the proposer, it will ask the mempool for the new block again, and the mempool may return a +different block $Z$, and we can have the same round as [Round 0](#round-0-1) just for a different block. As +a result, process $p$ calls `PrepareProposal` again but for a different value. When it reaches round $r$ +some process will propose block $X$ and if $p$ receives $2f+1$ $Precommit$ messages, it will decide on this +value. + + +## Scenario 3 + +$p$ calls `PrepareProposal` and `ProcessProposal` for many values, but decides on a value for which it did +not call `PrepareProposal` or `ProcessProposal`. + +In this scenario, in all rounds before $r$ we can have any round presented in [Scenario 1](#scenario-1) or +[Scenario 2](#scenario-2). What is important is that: + +* no proposer proposed block $X$ or if it did, process $p$, due to asynchrony, did not receive it in time, +so it did not call `ProcessProposal`, and + +* if $p$ was the proposer it proposed some other value $\neq X$. + +### Round $r$ + +1. **Propose:** A correct process is the proposer in this round, and it proposes block $X$. +Due to asynchrony, the proposal message arrives to process $p$ after its $timeoutPropose$ +expires and it sends $Prevote$ for $nil$. Consequently, process $p$ does not call +`ProcessProposal` for block $X$. However, the same proposal arrives at other processes +before their $timeoutPropose$ expires, and they send $Prevote$ for this proposal. +1. **Prevote:** Process $p$ receives $2f+1$ $Prevote$ messages for proposal $X$, updates correspondingly its +$validValue$ and $lockedValue$ and sends $Precommit$ message. All correct processes do the same. +1. **Precommit:** Finally, process $p$ receives $2f+1$ $Precommit$ messages, and decides on block $X$. + + + +## Scenario 4 + +[Scenario 3](#scenario-3) can be translated into a scenario where $p$ does not call `PrepareProposal` and +`ProcessProposal` at all. For this, it is necessary that process $p$ is not the proposer in any of the +rounds $0 <= r' <= r$ and that due to network asynchrony or Byzantine proposer, it does not receive the +proposal before $timeoutPropose$ expires. As a result, it will enter round $r$ without calling +`PrepareProposal` and `ProcessProposal` before it, and as shown in Round $r$ of [Scenario 3](#scenario-3) it +will decide in this round. Again without calling any of these two calls. diff --git a/cometbft/v0.38/spec/abci/Methods.mdx b/cometbft/v0.38/spec/abci/Methods.mdx new file mode 100644 index 00000000..29d24e21 --- /dev/null +++ b/cometbft/v0.38/spec/abci/Methods.mdx @@ -0,0 +1,911 @@ +--- +order: 2 +title: Methods +--- + +# Methods + +## Methods existing in ABCI + +### Echo + +* **Request**: + * `Message (string)`: A string to echo back +* **Response**: + * `Message (string)`: The input string +* **Usage**: + * Echo a string to test an ABCI client/server implementation + +### Flush + +* **Usage**: + * Signals that messages queued on the client should be flushed to + the server. It is called periodically by the client + implementation to ensure asynchronous requests are actually + sent, and is called immediately to make a synchronous request, + which returns when the Flush response comes back. + +### Info + +* **Request**: + + | Name | Type | Description | Field Number | + |---------------|--------|----------------------------------------|--------------| + | version | string | The CometBFT software semantic version | 1 | + | block_version | uint64 | The CometBFT Block version | 2 | + | p2p_version | uint64 | The CometBFT P2P version | 3 | + | abci_version | string | The CometBFT ABCI semantic version | 4 | + +* **Response**: + + | Name | Type | Description | Field Number | Deterministic | + |---------------------|--------|-----------------------------------------------------|--------------|---------------| + | data | string | Some arbitrary information | 1 | N/A | + | version | string | The application software semantic version | 2 | N/A | + | app_version | uint64 | The application version | 3 | N/A | + | last_block_height | int64 | Latest height for which the app persisted its state | 4 | N/A | + | last_block_app_hash | bytes | Latest AppHash returned by `FinalizeBlock` | 5 | N/A | + +* **Usage**: + * Return information about the application state. + * Used to sync CometBFT with the application during a handshake + that happens on startup or on recovery. + * The returned `app_version` will be included in the Header of every block. + * CometBFT expects `last_block_app_hash` and `last_block_height` to + be updated and persisted during `Commit`. + +> Note: Semantic version is a reference to [semantic versioning](https://semver.org/). Semantic versions in info will be displayed as X.X.x. + +### InitChain + +* **Request**: + + | Name | Type | Description | Field Number | + |------------------|-------------------------------------------------|-----------------------------------------------------|--------------| + | time | [google.protobuf.Timestamp][protobuf-timestamp] | Genesis time | 1 | + | chain_id | string | ID of the blockchain. | 2 | + | consensus_params | [ConsensusParams](#consensusparams) | Initial consensus-critical parameters. | 3 | + | validators | repeated [ValidatorUpdate](#validatorupdate) | Initial genesis validators, sorted by voting power. | 4 | + | app_state_bytes | bytes | Serialized initial application state. JSON bytes. | 5 | + | initial_height | int64 | Height of the initial block (typically `1`). | 6 | + +* **Response**: + + | Name | Type | Description | Field Number | Deterministic | + |------------------|----------------------------------------------|--------------------------------------------------|--------------|---------------| + | consensus_params | [ConsensusParams](#consensusparams) | Initial consensus-critical parameters (optional) | 1 | Yes | + | validators | repeated [ValidatorUpdate](#validatorupdate) | Initial validator set (optional). | 2 | Yes | + | app_hash | bytes | Initial application hash. | 3 | Yes | + +* **Usage**: + * Called once upon genesis. + * If `ResponseInitChain.Validators` is empty, the initial validator set will be the `RequestInitChain.Validators` + * If `ResponseInitChain.Validators` is not empty, it will be the initial + validator set (regardless of what is in `RequestInitChain.Validators`). + * This allows the app to decide if it wants to accept the initial validator + set proposed by CometBFT (ie. in the genesis file), or if it wants to use + a different one (perhaps computed based on some application specific + information in the genesis file). + * Both `RequestInitChain.Validators` and `ResponseInitChain.Validators` are [ValidatorUpdate](#validatorupdate) structs. + So, technically, they both are _updating_ the set of validators from the empty set. + +### Query + +* **Request**: + + | Name | Type | Description | Field Number | + |--------|--------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------| + | data | bytes | Request parameters for the application to interpret analogously to a [URI query component](https://www.rfc-editor.org/rfc/rfc3986#section-3.4). Can be used with or in lieu of `path`. | 1 | + | path | string | A request path for the application to interpret analogously to a [URI path component](https://www.rfc-editor.org/rfc/rfc3986#section-3.3) in e.g. routing. Can be used with or in lieu of `data`. Applications MUST interpret "/store" or any path starting with "/store/" as a query by key on the underlying store, in which case a key SHOULD be specified in `data`. Applications SHOULD allow queries over specific types like `/accounts/...` or `/votes/...`. | 2 | + | height | int64 | The block height against which to query (default=0 returns data for the latest committed block). Note that this is the height of the block containing the application's Merkle root hash, which represents the state as it was after committing the block at Height-1. | 3 | + | prove | bool | Return Merkle proof with response if possible. | 4 | + +* **Response**: + + | Name | Type | Description | Field Number | Deterministic | + |-----------|-----------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------|---------------| + | code | uint32 | Response code. | 1 | N/A | + | log | string | The output of the application's logger. | 3 | N/A | + | info | string | Additional information. | 4 | N/A | + | index | int64 | The index of the key in the tree. | 5 | N/A | + | key | bytes | The key of the matching data. | 6 | N/A | + | value | bytes | The value of the matching data. | 7 | N/A | + | proof_ops | [ProofOps](#proofops) | Serialized proof for the value data, if requested, to be verified against the `app_hash` for the given Height. | 8 | N/A | + | height | int64 | The block height from which data was derived. Note that this is the height of the block containing the application's Merkle root hash, which represents the state as it was after committing the block at Height-1 | 9 | N/A | + | codespace | string | Namespace for the `code`. | 10 | N/A | + +* **Usage**: + * Query for data from the application at current or past height. + * Optionally return Merkle proof. + * Merkle proof includes self-describing `type` field to support many types + of Merkle trees and encoding formats. + +### CheckTx + +* **Request**: + + | Name | Type | Description | Field Number | + |------|-------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------| + | tx | bytes | The request transaction bytes | 1 | + | type | CheckTxType | One of `CheckTx_New` or `CheckTx_Recheck`. `CheckTx_New` is the default and means that a full check of the tranasaction is required. `CheckTx_Recheck` types are used when the mempool is initiating a normal recheck of a transaction. | 2 | + +* **Response**: + + | Name | Type | Description | Field Number | Deterministic | + |------------|---------------------------------------------------|----------------------------------------------------------------------|--------------|---------------| + | code | uint32 | Response code. | 1 | N/A | + | data | bytes | Result bytes, if any. | 2 | N/A | + | log | string | The output of the application's logger. | 3 | N/A | + | info | string | Additional information. | 4 | N/A | + | gas_wanted | int64 | Amount of gas requested for transaction. | 5 | N/A | + | gas_used | int64 | Amount of gas consumed by transaction. | 6 | N/A | + | events | repeated [Event](/cometbft/v0.38/spec/abci/Outline#events) | Type & Key-Value events for indexing transactions (e.g. by account). | 7 | N/A | + | codespace | string | Namespace for the `code`. | 8 | N/A | + +* **Usage**: + + * Technically optional - not involved in processing blocks. + * Guardian of the mempool: every node runs `CheckTx` before letting a + transaction into its local mempool. + * The transaction may come from an external user or another node + * `CheckTx` validates the transaction against the current state of the application, + for example, checking signatures and account balances, but does not apply any + of the state changes described in the transaction. + * Transactions where `ResponseCheckTx.Code != 0` will be rejected - they will not be broadcast + to other nodes or included in a proposal block. + CometBFT attributes no other value to the response code. + +### Commit + +#### Parameters and Types + +* **Request**: + + Commit signals the application to persist application state. It takes no parameters. + +* **Response**: + + | Name | Type | Description | Field Number | Deterministic | + |---------------|-------|------------------------------------------------------------------------|--------------|---------------| + | retain_height | int64 | Blocks below this height may be removed. Defaults to `0` (retain all). | 3 | No | + +* **Usage**: + + * Signal the Application to persist the application state. + Application is expected to persist its state at the end of this call, before calling `ResponseCommit`. + * Use `ResponseCommit.retain_height` with caution! If all nodes in the network remove historical + blocks then this data is permanently lost, and no new nodes will be able to join the network and + bootstrap, unless state sync is enabled on the chain. Historical blocks may also be required for other purposes, e.g. auditing, replay of + non-persisted heights, light client verification, and so on. + +### ListSnapshots + +* **Request**: + + Empty request asking the application for a list of snapshots. + +* **Response**: + + | Name | Type | Description | Field Number | Deterministic | + |-----------|--------------------------------|--------------------------------|--------------|---------------| + | snapshots | repeated [Snapshot](#snapshot) | List of local state snapshots. | 1 | N/A | + +* **Usage**: + * Used during state sync to discover available snapshots on peers. + * See `Snapshot` data type for details. + +### LoadSnapshotChunk + +* **Request**: + + | Name | Type | Description | Field Number | + |--------|--------|-----------------------------------------------------------------------|--------------| + | height | uint64 | The height of the snapshot the chunk belongs to. | 1 | + | format | uint32 | The application-specific format of the snapshot the chunk belongs to. | 2 | + | chunk | uint32 | The chunk index, starting from `0` for the initial chunk. | 3 | + +* **Response**: + + | Name | Type | Description | Field Number | Deterministic | + |-------|-------|--------------------------------------------------------------------------------------------------------------------------------------------------------|--------------|---------------| + | chunk | bytes | The binary chunk contents, in an arbitrary format. Chunk messages cannot be larger than 16 MB _including metadata_, so 10 MB is a good starting point. | 1 | N/A | + +* **Usage**: + * Used during state sync to retrieve snapshot chunks from peers. + +### OfferSnapshot + +* **Request**: + + | Name | Type | Description | Field Number | + |----------|-----------------------|--------------------------------------------------------------------------|--------------| + | snapshot | [Snapshot](#snapshot) | The snapshot offered for restoration. | 1 | + | app_hash | bytes | The light client-verified app hash for this height, from the blockchain. | 2 | + +* **Response**: + + | Name | Type | Description | Field Number | Deterministic | + |--------|-------------------|-----------------------------------|--------------|---------------| + | result | [Result](#result) | The result of the snapshot offer. | 1 | N/A | + +#### Result + +```protobuf + enum Result { + UNKNOWN = 0; // Unknown result, abort all snapshot restoration + ACCEPT = 1; // Snapshot is accepted, start applying chunks. + ABORT = 2; // Abort snapshot restoration, and don't try any other snapshots. + REJECT = 3; // Reject this specific snapshot, try others. + REJECT_FORMAT = 4; // Reject all snapshots with this `format`, try others. + REJECT_SENDER = 5; // Reject all snapshots from all senders of this snapshot, try others. + } +``` + +* **Usage**: + * `OfferSnapshot` is called when bootstrapping a node using state sync. The application may + accept or reject snapshots as appropriate. Upon accepting, CometBFT will retrieve and + apply snapshot chunks via `ApplySnapshotChunk`. The application may also choose to reject a + snapshot in the chunk response, in which case it should be prepared to accept further + `OfferSnapshot` calls. + * Only `AppHash` can be trusted, as it has been verified by the light client. Any other data + can be spoofed by adversaries, so applications should employ additional verification schemes + to avoid denial-of-service attacks. The verified `AppHash` is automatically checked against + the restored application at the end of snapshot restoration. + * For more information, see the `Snapshot` data type or the [state sync section](/cometbft/v0.38/spec/p2p/legacy-docs/messages/state-sync). + +### ApplySnapshotChunk + +* **Request**: + + | Name | Type | Description | Field Number | + |--------|--------|---------------------------------------------------------------------------|--------------| + | index | uint32 | The chunk index, starting from `0`. CometBFT applies chunks sequentially. | 1 | + | chunk | bytes | The binary chunk contents, as returned by `LoadSnapshotChunk`. | 2 | + | sender | string | The P2P ID of the node who sent this chunk. | 3 | + +* **Response**: + + | Name | Type | Description | Field Number | Deterministic | + |----------------|---------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------|---------------| + | result | Result (see below) | The result of applying this chunk. | 1 | N/A | + | refetch_chunks | repeated uint32 | Refetch and reapply the given chunks, regardless of `result`. Only the listed chunks will be refetched, and reapplied in sequential order. | 2 | N/A | + | reject_senders | repeated string | Reject the given P2P senders, regardless of `Result`. Any chunks already applied will not be refetched unless explicitly requested, but queued chunks from these senders will be discarded, and new chunks or other snapshots rejected. | 3 | N/A | + +```proto + enum Result { + UNKNOWN = 0; // Unknown result, abort all snapshot restoration + ACCEPT = 1; // The chunk was accepted. + ABORT = 2; // Abort snapshot restoration, and don't try any other snapshots. + RETRY = 3; // Reapply this chunk, combine with `RefetchChunks` and `RejectSenders` as appropriate. + RETRY_SNAPSHOT = 4; // Restart this snapshot from `OfferSnapshot`, reusing chunks unless instructed otherwise. + REJECT_SNAPSHOT = 5; // Reject this snapshot, try a different one. + } +``` + +* **Usage**: + * The application can choose to refetch chunks and/or ban P2P peers as appropriate. CometBFT + will not do this unless instructed by the application. + * The application may want to verify each chunk, e.g. by attaching chunk hashes in + `Snapshot.Metadata` and/or incrementally verifying contents against `AppHash`. + * When all chunks have been accepted, CometBFT will make an ABCI `Info` call to verify that + `LastBlockAppHash` and `LastBlockHeight` matches the expected values, and record the + `AppVersion` in the node state. It then switches to block sync or consensus and joins the + network. + * If CometBFT is unable to retrieve the next chunk after some time (e.g. because no suitable + peers are available), it will reject the snapshot and try a different one via `OfferSnapshot`. + The application should be prepared to reset and accept it or abort as appropriate. + +## New methods introduced in ABCI 2.0 + +### PrepareProposal + +#### Parameters and Types + +* **Request**: + + | Name | Type | Description | Field Number | + |----------------------|-------------------------------------------------|-----------------------------------------------------------------------------------------------|--------------| + | max_tx_bytes | int64 | Currently configured maximum size in bytes taken by the modified transactions. | 1 | + | txs | repeated bytes | Preliminary list of transactions that have been picked as part of the block to propose. | 2 | + | local_last_commit | [ExtendedCommitInfo](#extendedcommitinfo) | Info about the last commit, obtained locally from CometBFT's data structures. | 3 | + | misbehavior | repeated [Misbehavior](#misbehavior) | List of information about validators that misbehaved. | 4 | + | height | int64 | The height of the block that will be proposed. | 5 | + | time | [google.protobuf.Timestamp][protobuf-timestamp] | Timestamp of the block that that will be proposed. | 6 | + | next_validators_hash | bytes | Merkle root of the next validator set. | 7 | + | proposer_address | bytes | [Address](../core/data_structures.md#address) of the validator that is creating the proposal. | 8 | + +* **Response**: + + | Name | Type | Description | Field Number | Deterministic | + |------|----------------|---------------------------------------------------------------------------------------------|--------------|---------------| + | txs | repeated bytes | Possibly modified list of transactions that have been picked as part of the proposed block. | 2 | No | + +* **Usage**: + * `RequestPrepareProposal`'s parameters `txs`, `misbehavior`, `height`, `time`, + `next_validators_hash`, and `proposer_address` are the same as in `RequestProcessProposal` + and `RequestFinalizeBlock`. + * `RequestPrepareProposal.local_last_commit` is a set of the precommit votes for the previous + height, including the ones that led to the decision of the previous block, + together with their corresponding vote extensions. + * The `height`, `time`, and `proposer_address` values match the values from the header of the + proposed block. + * `RequestPrepareProposal` contains a preliminary set of transactions `txs` that CometBFT + retrieved from the mempool, called _raw proposal_. The Application can modify this + set and return a modified set of transactions via `ResponsePrepareProposal.txs` . + * The Application _can_ modify the raw proposal: it can reorder, remove or add transactions. + Let `tx` be a transaction in `txs` (set of transactions within `RequestPrepareProposal`): + * If the Application considers that `tx` should not be proposed in this block, e.g., + there are other transactions with higher priority, then it should not include it in + `ResponsePrepareProposal.txs`. However, this will not remove `tx` from the mempool. + * If the Application wants to add a new transaction to the proposed block, then the + Application includes it in `ResponsePrepareProposal.txs`. CometBFT will not add + the transaction to the mempool. + * The Application should be aware that removing and adding transactions may compromise + _traceability_. + > Consider the following example: the Application transforms a client-submitted + transaction `t1` into a second transaction `t2`, i.e., the Application asks CometBFT + to remove `t1` from the block and add `t2` to the block. If a client wants to eventually check what + happened to `t1`, it will discover that `t1` is not in a + committed block (assuming a _re-CheckTx_ evicted it from the mempool), getting the wrong idea that `t1` did not make it into a block. Note + that `t2` _will be_ in a committed block, but unless the Application tracks this + information, no component will be aware of it. Thus, if the Application wants + traceability, it is its responsibility's to support it. For instance, the Application + could attach to a transformed transaction a list with the hashes of the transactions it + derives from. + * The Application MAY configure CometBFT to include a list of transactions in `RequestPrepareProposal.txs` + whose total size in bytes exceeds `RequestPrepareProposal.max_tx_bytes`. + If the Application sets `ConsensusParams.Block.MaxBytes` to -1, CometBFT + will include _all_ transactions currently in the mempool in `RequestPrepareProposal.txs`, + which may not fit in `RequestPrepareProposal.max_tx_bytes`. + Therefore, if the size of `RequestPrepareProposal.txs` is greater than + `RequestPrepareProposal.max_tx_bytes`, the Application MUST remove transactions to ensure + that the `RequestPrepareProposal.max_tx_bytes` limit is respected by those transactions + returned in `ResponsePrepareProposal.txs`. + This is specified in [Requirement 2](/cometbft/v0.38/spec/abci/Requirements-for-the-Application). + * As a result of executing the prepared proposal, the Application may produce block events or transaction events. + The Application must keep those events until a block is decided and then pass them on to CometBFT via + `ResponseFinalizeBlock`. + * CometBFT does NOT provide any additional validity checks (such as checking for duplicate + transactions). + {/* + As a sanity check, CometBFT will check the returned parameters for validity if the Application modified them. + In particular, `ResponsePrepareProposal.txs` will be deemed invalid if there are duplicate transactions in the list. + */} + * If CometBFT fails to validate the `ResponsePrepareProposal`, CometBFT will assume the + Application is faulty and crash. + * The implementation of `PrepareProposal` MAY be non-deterministic. + + +#### When does CometBFT call "PrepareProposal" ? + +When a validator _p_ enters consensus round _r_, height _h_, in which _p_ is the proposer, +and _p_'s _validValue_ is `nil`: + +1. CometBFT collects outstanding transactions from _p_'s mempool + * the transactions will be collected in order of priority + * _p_'s CometBFT creates a block header. +2. _p_'s CometBFT calls `RequestPrepareProposal` with the newly generated block, the local + commit of the previous height (with vote extensions), and any outstanding evidence of + misbehavior. The call is synchronous: CometBFT's execution will block until the Application + returns from the call. +3. The Application uses the information received (transactions, commit info, misbehavior, time) to + (potentially) modify the proposal. + * the Application MAY fully execute the block and produce a candidate state (immediate execution) + * the Application can manipulate transactions: + * leave transactions untouched + * add new transactions (not present initially) to the proposal + * remove transactions from the proposal (but not from the mempool thus effectively _delaying_ them) - the + Application does not include the transaction in `ResponsePrepareProposal.txs`. + * modify transactions (e.g. aggregate them). As explained above, this compromises client traceability, unless + it is implemented at the Application level. + * reorder transactions - the Application reorders transactions in the list + * the Application MAY use the vote extensions in the commit info to modify the proposal, in which case it is suggested + that extensions be validated in the same maner as done in `VerifyVoteExtension`, since extensions of votes included + in the commit info after the minimum of +2/3 had been reached are not verified. +4. The Application includes the transaction list (whether modified or not) in the return parameters + (see the rules in section _Usage_), and returns from the call. +5. _p_ uses the (possibly) modified block as _p_'s proposal in round _r_, height _h_. + +Note that, if _p_ has a non-`nil` _validValue_ in round _r_, height _h_, +the consensus algorithm will use it as proposal and will not call `RequestPrepareProposal`. + +### ProcessProposal + +#### Parameters and Types + +* **Request**: + + | Name | Type | Description | Field Number | + |----------------------|-------------------------------------------------|-------------------------------------------------------------------------------------------|--------------| + | txs | repeated bytes | List of transactions of the proposed block. | 1 | + | proposed_last_commit | [CommitInfo](#commitinfo) | Info about the last commit, obtained from the information in the proposed block. | 2 | + | misbehavior | repeated [Misbehavior](#misbehavior) | List of information about validators that misbehaved. | 3 | + | hash | bytes | The hash of the proposed block. | 4 | + | height | int64 | The height of the proposed block. | 5 | + | time | [google.protobuf.Timestamp][protobuf-timestamp] | Timestamp of the proposed block. | 6 | + | next_validators_hash | bytes | Merkle root of the next validator set. | 7 | + | proposer_address | bytes | [Address](/cometbft/v0.38/spec/core/Data_structures#address) of the validator that created the proposal. | 8 | + +* **Response**: + + | Name | Type | Description | Field Number | Deterministic | + |--------|-----------------------------------|------------------------------------------------------------------|--------------|---------------| + | status | [ProposalStatus](#proposalstatus) | `enum` that signals if the application finds the proposal valid. | 1 | Yes | + +* **Usage**: + * Contains all information on the proposed block needed to fully execute it. + * The Application may fully execute the block as though it was handling + `RequestFinalizeBlock`. + * However, any resulting state changes must be kept as _candidate state_, + and the Application should be ready to discard it in case another block is decided. + * `RequestProcessProposal` is also called at the proposer of a round. + Normally the call to `RequestProcessProposal` occurs right after the call to `RequestPrepareProposal` and + `RequestProcessProposal` matches the block produced based on `ResponsePrepareProposal` (i.e., + `RequestPrepareProposal.txs` equals `RequestProcessProposal.txs`). + However, no such guarantee is made since, in the presence of failures, `RequestProcessProposal` may match + `ResponsePrepareProposal` from an earlier invocation or `ProcessProposal` may not be invoked at all. + * The height and time values match the values from the header of the proposed block. + * If `ResponseProcessProposal.status` is `REJECT`, consensus assumes the proposal received + is not valid. + * The Application MAY fully execute the block (immediate execution) + * The implementation of `ProcessProposal` MUST be deterministic. Moreover, the value of + `ResponseProcessProposal.status` MUST **exclusively** depend on the parameters passed in + the call to `RequestProcessProposal`, and the last committed Application state + (see [Requirements](/cometbft/v0.38/spec/abci/Requirements-for-the-Application) section). + * Moreover, application implementors SHOULD always set `ResponseProcessProposal.status` to `ACCEPT`, + unless they _really_ know what the potential liveness implications of returning `REJECT` are. + +#### When does CometBFT call "ProcessProposal" ? + +When a node _p_ enters consensus round _r_, height _h_, in which _q_ is the proposer (possibly _p_ = _q_): + +1. _p_ sets up timer `ProposeTimeout`. +2. If _p_ is the proposer, _p_ executes steps 1-6 in [PrepareProposal](#prepareproposal). +3. Upon reception of Proposal message (which contains the header) for round _r_, height _h_ from + _q_, _p_ verifies the block header. +4. Upon reception of Proposal message, along with all the block parts, for round _r_, height _h_ + from _q_, _p_ follows the validators' algorithm to check whether it should prevote for the + proposed block, or `nil`. +5. If the validators' consensus algorithm indicates _p_ should prevote non-nil: + 1. CometBFT calls `RequestProcessProposal` with the block. The call is synchronous. + 2. The Application checks/processes the proposed block, which is read-only, and returns + `ACCEPT` or `REJECT` in the `ResponseProcessProposal.status` field. + * The Application, depending on its needs, may call `ResponseProcessProposal` + * either after it has completely processed the block (immediate execution), + * or after doing some basic checks, and process the block asynchronously. In this case the + Application will not be able to reject the block, or force prevote/precommit `nil` + afterwards. + * or immediately, returning `ACCEPT`, if _p_ is not a validator + and the Application does not want non-validating nodes to handle `ProcessProposal` + 3. If _p_ is a validator and the returned value is + * `ACCEPT`: _p_ prevotes on this proposal for round _r_, height _h_. + * `REJECT`: _p_ prevotes `nil`. + * + +### ExtendVote + +#### Parameters and Types + +* **Request**: + + | Name | Type | Description | Field Number | + |----------------------|-------------------------------------------------|-------------------------------------------------------------------------------------------|--------------| + | hash | bytes | The header hash of the proposed block that the vote extension is to refer to. | 1 | + | height | int64 | Height of the proposed block (for sanity check). | 2 | + | time | [google.protobuf.Timestamp][protobuf-timestamp] | Timestamp of the proposed block (that the extension is to refer to). | 3 | + | txs | repeated bytes | List of transactions of the block that the extension is to refer to. | 4 | + | proposed_last_commit | [CommitInfo](#commitinfo) | Info about the last proposed block's last commit. | 5 | + | misbehavior | repeated [Misbehavior](#misbehavior) | List of information about validators that misbehaved contained in the proposed block. | 6 | + | next_validators_hash | bytes | Merkle root of the next validator set contained in the proposed block. | 7 | + | proposer_address | bytes | [Address](/cometbft/v0.38/spec/core/Data_structures#address) of the validator that created the proposal. | 8 | + +* **Response**: + + | Name | Type | Description | Field Number | Deterministic | + |----------------|-------|-------------------------------------------------------|--------------|---------------| + | vote_extension | bytes | Information signed by by CometBFT. Can have 0 length. | 1 | No | + +* **Usage**: + * `ResponseExtendVote.vote_extension` is application-generated information that will be signed + by CometBFT and attached to the Precommit message. + * The Application may choose to use an empty vote extension (0 length). + * The contents of `RequestExtendVote` correspond to the proposed block on which the consensus algorithm + will send the Precommit message. + * `ResponseExtendVote.vote_extension` will only be attached to a non-`nil` Precommit message. If the consensus algorithm is to + precommit `nil`, it will not call `RequestExtendVote`. + * The Application logic that creates the extension can be non-deterministic. + +#### When does CometBFT call `ExtendVote`? + +When a validator _p_ is in consensus state _prevote_ of round _r_, height _h_, in which _q_ is the proposer; and _p_ has received + +* the Proposal message _v_ for round _r_, height _h_, along with all the block parts, from _q_, +* `Prevote` messages from _2f + 1_ validators' voting power for round _r_, height _h_, prevoting for the same block _id(v)_, + +then _p_ locks _v_ and sends a Precommit message in the following way + +1. _p_ sets _lockedValue_ and _validValue_ to _v_, and sets _lockedRound_ and _validRound_ to _r_ +2. _p_'s CometBFT calls `RequestExtendVote` with _v_ (`RequestExtendVote`). The call is synchronous. +3. The Application returns an array of bytes, `ResponseExtendVote.extension`, which is not interpreted by the consensus algorithm. +4. _p_ sets `ResponseExtendVote.extension` as the value of the `extension` field of type + [CanonicalVoteExtension](/cometbft/v0.38/spec/core/Data_structures#canonicalvoteextension), + populates the other fields in [CanonicalVoteExtension](/cometbft/v0.38/spec/core/Data_structures#canonicalvoteextension), + and signs the populated data structure. +5. _p_ constructs and signs the [CanonicalVote](/cometbft/v0.38/spec/core/Data_structures#canonicalvote) structure. +6. _p_ constructs the Precommit message (i.e. [Vote](/cometbft/v0.38/spec/core/Data_structures#vote) structure) + using [CanonicalVoteExtension](../core/data_structures.md#canonicalvoteextension) + and [CanonicalVote](/cometbft/v0.38/spec/core/Data_structures#canonicalvoteextension). +7. _p_ broadcasts the Precommit message. + +In the cases when _p_ is to broadcast `precommit nil` messages (either _2f+1_ `prevote nil` messages received, +or _timeoutPrevote_ triggered), _p_'s CometBFT does **not** call `RequestExtendVote` and will not include +a [CanonicalVoteExtension](../core/data_structures.md#canonicalvoteextension) field in the `precommit nil` message. + +### VerifyVoteExtension + +#### Parameters and Types + +* **Request**: + + | Name | Type | Description | Field Number | + |-------------------|-------|-------------------------------------------------------------------------------------------|--------------| + | hash | bytes | The hash of the proposed block that the vote extension refers to. | 1 | + | validator_address | bytes | [Address](/cometbft/v0.38/spec/core/Data_structures#address) of the validator that signed the extension. | 2 | + | height | int64 | Height of the block (for sanity check). | 3 | + | vote_extension | bytes | Application-specific information signed by CometBFT. Can have 0 length. | 4 | + +* **Response**: + + | Name | Type | Description | Field Number | Deterministic | + |--------|-------------------------------|----------------------------------------------------------------|--------------|---------------| + | status | [VerifyStatus](#verifystatus) | `enum` signaling if the application accepts the vote extension | 1 | Yes | + +* **Usage**: + * `RequestVerifyVoteExtension.vote_extension` can be an empty byte array. The Application's + interpretation of it should be + that the Application running at the process that sent the vote chose not to extend it. + CometBFT will always call `RequestVerifyVoteExtension`, even for 0 length vote extensions. + * `RequestVerifyVoteExtension` is not called for precommit votes sent by the local process. + * `RequestVerifyVoteExtension.hash` refers to a proposed block. There is not guarantee that + this proposed block has previously been exposed to the Application via `ProcessProposal`. + * If `ResponseVerifyVoteExtension.status` is `REJECT`, the consensus algorithm will reject the whole received vote. + See the [Requirements](/cometbft/v0.38/spec/abci/Requirements-for-the-Application) section to understand the potential + liveness implications of this. + * The implementation of `VerifyVoteExtension` MUST be deterministic. Moreover, the value of + `ResponseVerifyVoteExtension.status` MUST **exclusively** depend on the parameters passed in + the call to `RequestVerifyVoteExtension`, and the last committed Application state + (see [Requirements](/cometbft/v0.38/spec/abci/Requirements-for-the-Application) section). + * Moreover, application implementers SHOULD always set `ResponseVerifyVoteExtension.status` to `ACCEPT`, + unless they _really_ know what the potential liveness implications of returning `REJECT` are. + +#### When does CometBFT call `VerifyVoteExtension`? + +When a node _p_ is in consensus round _r_, height _h_, and _p_ receives a Precommit +message for round _r_, height _h_ from validator _q_ (_q_ ≠ _p_): + +1. If the Precommit message does not contain a vote extension with a valid signature, _p_ + discards the Precommit message as invalid. + * a 0-length vote extension is valid as long as its accompanying signature is also valid. +2. Else, _p_'s CometBFT calls `RequestVerifyVoteExtension`. +3. The Application returns `ACCEPT` or `REJECT` via `ResponseVerifyVoteExtension.status`. +4. If the Application returns + * `ACCEPT`, _p_ will keep the received vote, together with its corresponding + vote extension in its internal data structures. It will be used to populate the [ExtendedCommitInfo](#extendedcommitinfo) + structure in calls to `RequestPrepareProposal`, in rounds of height _h + 1_ where _p_ is the proposer. + * `REJECT`, _p_ will deem the Precommit message invalid and discard it. + +When a node _p_ is in consensus round _0_, height _h_, and _p_ receives a Precommit +message for CommitRound _r_, height _h-1_ from validator _q_ (_q_ ≠ _p_), _p_ +MAY add the Precommit message and associated extension to [ExtendedCommitInfo](#extendedcommitinfo) +without calling `RequestVerifyVoteExtension` to verify it. + + +### FinalizeBlock + +#### Parameters and Types + +* **Request**: + + | Name | Type | Description | Field Number | + |----------------------|-------------------------------------------------|-------------------------------------------------------------------------------------------|--------------| + | txs | repeated bytes | List of transactions committed as part of the block. | 1 | + | decided_last_commit | [CommitInfo](#commitinfo) | Info about the last commit, obtained from the block that was just decided. | 2 | + | misbehavior | repeated [Misbehavior](#misbehavior) | List of information about validators that misbehaved. | 3 | + | hash | bytes | The block's hash. | 4 | + | height | int64 | The height of the finalized block. | 5 | + | time | [google.protobuf.Timestamp][protobuf-timestamp] | Timestamp of the finalized block. | 6 | + | next_validators_hash | bytes | Merkle root of the next validator set. | 7 | + | proposer_address | bytes | [Address](/cometbft/v0.38/spec/core/Data_structures#address) of the validator that created the proposal. | 8 | + +* **Response**: + + | Name | Type | Description | Field Number | Deterministic | + |-------------------------|---------------------------------------------------|----------------------------------------------------------------------------------|--------------|---------------| + | events | repeated [Event](/cometbft/v0.38/spec/abci/Outline#events) | Type & Key-Value events for indexing | 1 | No | + | tx_results | repeated [ExecTxResult](#exectxresult) | List of structures containing the data resulting from executing the transactions | 2 | Yes | + | validator_updates | repeated [ValidatorUpdate](#validatorupdate) | Changes to validator set (set voting power to 0 to remove). | 3 | Yes | + | consensus_param_updates | [ConsensusParams](#consensusparams) | Changes to gas, size, and other consensus-related parameters. | 4 | Yes | + | app_hash | bytes | The Merkle root hash of the application state. | 5 | Yes | + +* **Usage**: + * Contains the fields of the newly decided block. + * This method is equivalent to the call sequence `BeginBlock`, [`DeliverTx`], + and `EndBlock` in ABCI 1.0. + * The height and time values match the values from the header of the proposed block. + * The Application can use `RequestFinalizeBlock.decided_last_commit` and `RequestFinalizeBlock.misbehavior` + to determine rewards and punishments for the validators. + * The Application executes the transactions in `RequestFinalizeBlock.txs` deterministically, + according to the rules set up by the Application, before returning control to CometBFT. + Alternatively, it can apply the candidate state corresponding to the same block previously + executed via `PrepareProposal` or `ProcessProposal`. + * `ResponseFinalizeBlock.tx_results[i].Code == 0` only if the _i_-th transaction is fully valid. + * The Application must provide values for `ResponseFinalizeBlock.app_hash`, + `ResponseFinalizeBlock.tx_results`, `ResponseFinalizeBlock.validator_updates`, and + `ResponseFinalizeBlock.consensus_param_updates` as a result of executing the block. + * The values for `ResponseFinalizeBlock.validator_updates`, or + `ResponseFinalizeBlock.consensus_param_updates` may be empty. In this case, CometBFT will keep + the current values. + * `ResponseFinalizeBlock.validator_updates`, triggered by block `H`, affect validation + for blocks `H+1`, `H+2`, and `H+3`. Heights following a validator update are affected in the following way: + * Height `H+1`: `NextValidatorsHash` includes the new `validator_updates` value. + * Height `H+2`: The validator set change takes effect and `ValidatorsHash` is updated. + * Height `H+3`: `*_last_commit` fields in `PrepareProposal`, `ProcessProposal`, and + `FinalizeBlock` now include the altered validator set. + * `ResponseFinalizeBlock.consensus_param_updates` returned for block `H` apply to the consensus + params for block `H+1`. For more information on the consensus parameters, + see the [consensus parameters](/cometbft/v0.38/spec/abci/Requirements-for-the-Application#consensus-parameters) + section. + * `ResponseFinalizeBlock.app_hash` contains an (optional) Merkle root hash of the application state. + * `ResponseFinalizeBlock.app_hash` is included as the `Header.AppHash` in the next block. + * `ResponseFinalizeBlock.app_hash` may also be empty or hard-coded, but MUST be + **deterministic** - it must not be a function of anything that did not come from the parameters + of `RequestFinalizeBlock` and the previous committed state. + * Later calls to `Query` can return proofs about the application state anchored + in this Merkle root hash. + * The implementation of `FinalizeBlock` MUST be deterministic, since it is + making the Application's state evolve in the context of state machine replication. + * Currently, CometBFT will fill up all fields in `RequestFinalizeBlock`, even if they were + already passed on to the Application via `RequestPrepareProposal` or `RequestProcessProposal`. + * When calling `FinalizeBlock` with a block, the consensus algorithm run by CometBFT guarantees + that at least one non-byzantine validator has run `ProcessProposal` on that block. + +#### When does CometBFT call `FinalizeBlock`? + +When a node _p_ is in consensus height _h_, and _p_ receives + +* the Proposal message with block _v_ for a round _r_, along with all its block parts, from _q_, + which is the proposer of round _r_, height _h_, +* `Precommit` messages from _2f + 1_ validators' voting power for round _r_, height _h_, + precommitting the same block _id(v)_, + +then _p_ decides block _v_ and finalizes consensus for height _h_ in the following way + +1. _p_ persists _v_ as the decision for height _h_. +2. _p_'s CometBFT calls `RequestFinalizeBlock` with _v_'s data. The call is synchronous. +3. _p_'s Application executes block _v_. +4. _p_'s Application calculates and returns the _AppHash_, along with a list containing + the outputs of each of the transactions executed. +5. _p_'s CometBFT hashes all the transaction outputs and stores it in _ResultHash_. +6. _p_'s CometBFT persists the transaction outputs, _AppHash_, and _ResultsHash_. +7. _p_'s CometBFT locks the mempool — no calls to `CheckTx` on new transactions. +8. _p_'s CometBFT calls `RequestCommit` to instruct the Application to persist its state. +9. _p_'s CometBFT, optionally, re-checks all outstanding transactions in the mempool + against the newly persisted Application state. +10. _p_'s CometBFT unlocks the mempool — newly received transactions can now be checked. +11. _p_ starts consensus for height _h+1_, round 0 + +## Data Types existing in ABCI + +Most of the data structures used in ABCI are shared [common data structures](/cometbft/v0.38/spec/core/Data_structures). In certain cases, ABCI uses different data structures which are documented here: + +### Validator + +* **Fields**: + + | Name | Type | Description | Field Number | + |---------|-------|------------------------------------------------------------|--------------| + | address | bytes | [Address](/cometbft/v0.38/spec/core/Data_structures#address) of validator | 1 | + | power | int64 | Voting power of the validator | 3 | + +* **Usage**: + * Validator identified by address + * Used as part of `VoteInfo` within `CommitInfo` (used in `ProcessProposal` + and `FinalizeBlock`), and `ExtendedCommitInfo` (used in `PrepareProposal`). + * Does not include PubKey to avoid sending potentially large quantum pubkeys + over the ABCI + +### ValidatorUpdate + +* **Fields**: + + | Name | Type | Description | Field Number | Deterministic | + |---------|--------------------------------------------------|-------------------------------|--------------|---------------| + | pub_key | [Public Key](/cometbft/v0.38/spec/core/Data_structures) | Public key of the validator | 1 | Yes | + | power | int64 | Voting power of the validator | 2 | Yes | + +* **Usage**: + * Validator identified by PubKey + * Used to tell CometBFT to update the validator set + +### Misbehavior + +* **Fields**: + + | Name | Type | Description | Field Number | + |--------------------|-------------------------------------------------|--------------------------------------------------------------|--------------| + | type | [MisbehaviorType](#misbehaviortype) | Type of the misbehavior. An enum of possible misbehaviors. | 1 | + | validator | [Validator](#validator) | The offending validator | 2 | + | height | int64 | Height when the offense occurred | 3 | + | time | [google.protobuf.Timestamp][protobuf-timestamp] | Timestamp of the block that was committed at height `height` | 4 | + | total_voting_power | int64 | Total voting power of the validator set at height `height` | 5 | + +#### MisbehaviorType + +* **Fields** + + MisbehaviorType is an enum with the listed fields: + + | Name | Field Number | + |---------------------|--------------| + | UNKNOWN | 0 | + | DUPLICATE_VOTE | 1 | + | LIGHT_CLIENT_ATTACK | 2 | + +### ConsensusParams + +* **Fields**: + + | Name | Type | Description | Field Number | Deterministic | + |-----------|---------------------------------------------------------------|------------------------------------------------------------------------------|--------------|---------------| + | block | [BlockParams](/cometbft/v0.38/spec/core/Data_structures#blockparams) | Parameters limiting the size of a block and time between consecutive blocks. | 1 | Yes | + | evidence | [EvidenceParams](/cometbft/v0.38/spec/core/Data_structures#evidenceparams) | Parameters limiting the validity of evidence of byzantine behaviour. | 2 | Yes | + | validator | [ValidatorParams](/cometbft/v0.38/spec/core/Data_structures#validatorparams) | Parameters limiting the types of public keys validators can use. | 3 | Yes | + | version | [VersionsParams](/cometbft/v0.38/spec/core/Data_structures#versionparams) | The ABCI application version. | 4 | Yes | + +### ProofOps + +* **Fields**: + + | Name | Type | Description | Field Number | Deterministic | + |------|------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------|---------------| + | ops | repeated [ProofOp](#proofop) | List of chained Merkle proofs, of possibly different types. The Merkle root of one op is the value being proven in the next op. The Merkle root of the final op should equal the ultimate root hash being verified against.. | 1 | N/A | + +### ProofOp + +* **Fields**: + + | Name | Type | Description | Field Number | Deterministic | + |------|--------|------------------------------------------------|--------------|---------------| + | type | string | Type of Merkle proof and how it's encoded. | 1 | N/A | + | key | bytes | Key in the Merkle tree that this proof is for. | 2 | N/A | + | data | bytes | Encoded Merkle proof for the key. | 3 | N/A | + +### Snapshot + +* **Fields**: + + | Name | Type | Description | Field Number | Deterministic | + |----------|--------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------|---------------| + | height | uint64 | The height at which the snapshot was taken (after commit). | 1 | N/A | + | format | uint32 | An application-specific snapshot format, allowing applications to version their snapshot data format and make backwards-incompatible changes. CometBFT does not interpret this. | 2 | N/A | + | chunks | uint32 | The number of chunks in the snapshot. Must be at least 1 (even if empty). | 3 | N/A | + | hash | bytes | An arbitrary snapshot hash. Must be equal only for identical snapshots across nodes. CometBFT does not interpret the hash, it only compares them. | 4 | N/A | + | metadata | bytes | Arbitrary application metadata, for example chunk hashes or other verification data. | 5 | N/A | + +* **Usage**: + * Used for state sync snapshots, see the [state sync section](/cometbft/v0.38/spec/p2p/legacy-docs/messages/state-sync) for details. + * A snapshot is considered identical across nodes only if _all_ fields are equal (including + `Metadata`). Chunks may be retrieved from all nodes that have the same snapshot. + * When sent across the network, a snapshot message can be at most 4 MB. + +## Data types introduced or modified in ABCI++ + +### VoteInfo + +* **Fields**: + + | Name | Type | Description | Field Number | + |---------------|-------------------------------------------------------|------------------------------------------------------------------------------------------|--------------| + | validator | [Validator](#validator) | The validator that sent the vote. | 1 | + | block_id_flag | [BlockIDFlag](../core/data_structures.md#blockidflag) | Indicates whether the validator voted the last block, nil, or its vote was not received. | 3 | + +* **Usage**: + * Indicates whether a validator signed the last block, allowing for rewards based on validator availability. + * This information is typically extracted from a proposed or decided block. + +### ExtendedVoteInfo + +* **Fields**: + + | Name | Type | Description | Field Number | + |---------------------|-------------------------------------------------------|---------------------------------------------------------------------------------------------|--------------| + | validator | [Validator](#validator) | The validator that sent the vote. | 1 | + | vote_extension | bytes | Non-deterministic extension provided by the sending validator's Application. | 3 | + | extension_signature | bytes | Signature of the vote extension produced by the sending validator and verified by CometBFT. | 4 | + | block_id_flag | [BlockIDFlag](../core/data_structures.md#blockidflag) | Indicates whether the validator voted the last block, nil, or its vote was not received. | 5 | + +* **Usage**: + * Indicates whether a validator signed the last block, allowing for rewards based on validator availability. + * This information is extracted from CometBFT's data structures in the local process. + * `vote_extension` contains the sending validator's vote extension, whose signature was verified by CometBFT. It can be empty. + * `extension_signature` is the signature of the vote extension, which was verified verified by CometBFT. This way, we expose the signature to the application for further processing or verification. + +### CommitInfo + +* **Fields**: + + | Name | Type | Description | Field Number | + |-------|--------------------------------|----------------------------------------------------------------------------------------------|--------------| + | round | int32 | Commit round. Reflects the round at which the block proposer decided in the previous height. | 1 | + | votes | repeated [VoteInfo](#voteinfo) | List of validators' addresses in the last validator set with their voting information. | 2 | + +* **Notes** + * The `VoteInfo` in `votes` are ordered by the voting power of the validators (descending order, highest to lowest voting power). + * CometBFT guarantees the `votes` ordering through its logic to update the validator set in which, in the end, the validators are sorted (descending) by their voting power. + * The ordering is also persisted when a validator set is saved in the store. + * The validator set is loaded from the store when building the `CommitInfo`, ensuring order is maintained from the persisted validator set. + +### ExtendedCommitInfo + +* **Fields**: + + | Name | Type | Description | Field Number | + |-------|------------------------------------------------|-------------------------------------------------------------------------------------------------------------------|--------------| + | round | int32 | Commit round. Reflects the round at which the block proposer decided in the previous height. | 1 | + | votes | repeated [ExtendedVoteInfo](#extendedvoteinfo) | List of validators' addresses in the last validator set with their voting information, including vote extensions. | 2 | + +* **Notes** + * The `ExtendedVoteInfo` in `votes` are ordered by the voting power of the validators (descending order, highest to lowest voting power). + * CometBFT guarantees the `votes` ordering through its logic to update the validator set in which, in the end, the validators are sorted (descending) by their voting power. + * The ordering is also persisted when a validator set is saved in the store. + * The validator set is loaded from the store when building the `ExtendedCommitInfo`, ensuring order is maintained from the persisted validator set. + +### ExecTxResult + +* **Fields**: + + | Name | Type | Description | Field Number | Deterministic | + |------------|---------------------------------------------------|----------------------------------------------------------------------|--------------|---------------| + | code | uint32 | Response code. | 1 | Yes | + | data | bytes | Result bytes, if any. | 2 | Yes | + | log | string | The output of the application's logger. | 3 | No | + | info | string | Additional information. | 4 | No | + | gas_wanted | int64 | Amount of gas requested for transaction. | 5 | Yes | + | gas_used | int64 | Amount of gas consumed by transaction. | 6 | Yes | + | events | repeated [Event](abci++_basic_concepts.md#events) | Type & Key-Value events for indexing transactions (e.g. by account). | 7 | No | + | codespace | string | Namespace for the `code`. | 8 | Yes | + +### ProposalStatus + +```proto +enum ProposalStatus { + UNKNOWN = 0; // Unknown status. Returning this from the application is always an error. + ACCEPT = 1; // Status that signals that the application finds the proposal valid. + REJECT = 2; // Status that signals that the application finds the proposal invalid. +} +``` + +* **Usage**: + * Used within the [ProcessProposal](#processproposal) response. + * If `Status` is `UNKNOWN`, a problem happened in the Application. CometBFT will assume the application is faulty and crash. + * If `Status` is `ACCEPT`, the consensus algorithm accepts the proposal and will issue a Prevote message for it. + * If `Status` is `REJECT`, the consensus algorithm rejects the proposal and will issue a Prevote for `nil` instead. + + +### VerifyStatus + +```proto +enum VerifyStatus { + UNKNOWN = 0; // Unknown status. Returning this from the application is always an error. + ACCEPT = 1; // Status that signals that the application finds the vote extension valid. + REJECT = 2; // Status that signals that the application finds the vote extension invalid. +} +``` + +* **Usage**: + * Used within the [VerifyVoteExtension](#verifyvoteextension) response. + * If `Status` is `UNKNOWN`, a problem happened in the Application. CometBFT will assume the application is faulty and crash. + * If `Status` is `ACCEPT`, the consensus algorithm will accept the vote as valid. + * If `Status` is `REJECT`, the consensus algorithm will reject the vote as invalid. + +[protobuf-timestamp]: https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#google.protobuf.Timestamp diff --git a/cometbft/v0.38/spec/abci/Outline.mdx b/cometbft/v0.38/spec/abci/Outline.mdx new file mode 100644 index 00000000..064e80b1 --- /dev/null +++ b/cometbft/v0.38/spec/abci/Outline.mdx @@ -0,0 +1,437 @@ +--- +order: 1 +title: Overview and basic concepts +--- +{/* trigger rebuild */} + +## Outline + +- [Overview and basic concepts](#overview-and-basic-concepts) + - [ABCI++ vs. ABCI](#abci-vs-abci) + - [Methods overview](#methods-overview) + - [Consensus/block execution methods](#consensusblock-execution-methods) + - [Mempool methods](#mempool-methods) + - [Info methods](#info-methods) + - [State-sync methods](#state-sync-methods) + - [Other methods](#other-methods) + - [Proposal timeout](#proposal-timeout) + - [Deterministic State-Machine Replication](#deterministic-state-machine-replication) + - [Events](#events) + - [Evidence](#evidence) + - [Errors](#errors) + - [`CheckTx`](#checktx) + - [`ExecTxResult` (as part of `FinalizeBlock`)](#exectxresult-as-part-of-finalizeblock) + - [`Query`](#query) + +# Overview and basic concepts + +## ABCI 2.0 vs. ABCI + +[↑ Back to Outline](#outline) + +The Application's main role is to execute blocks decided (a.k.a. finalized) by consensus. The +decided blocks are the consensus's main output to the (replicated) Application. With ABCI, the +application only interacts with consensus at *decision* time. This restricted mode of interaction +prevents numerous features for the Application, including many scalability improvements that are +now better understood than when ABCI was first written. For example, many ideas proposed to improve +scalability can be boiled down to "make the block proposers do work, so the network does not have +to". This includes optimizations such as transaction level signature aggregation, state transition +proofs, etc. Furthermore, many new security properties cannot be achieved in the current paradigm, +as the Application cannot require validators to do more than executing the transactions contained in +finalized blocks. This includes features such as threshold cryptography, and guaranteed IBC +connection attempts. + +ABCI 2.0 addresses these limitations by allowing the application to intervene at three key places of +consensus execution: (a) at the moment a new proposal is to be created, (b) at the moment a +proposal is to be validated, and (c) at the moment a (precommit) vote is sent/received. +The new interface allows block proposers to perform application-dependent +work in a block through the `PrepareProposal` method (a); and validators to perform application-dependent work +and checks in a proposed block through the `ProcessProposal` method (b); and applications to require their validators +to do more than just validate blocks through the `ExtendVote` and `VerifyVoteExtensions` methods (c). + +Furthermore, ABCI 2.0 coalesces {`BeginBlock`, [`DeliverTx`], `EndBlock`} into `FinalizeBlock`, as a +simplified, efficient way to deliver a decided block to the Application. + +## Methods overview + +[↑ Back to Outline](#outline) + +Methods can be classified into four categories: *consensus*, *mempool*, *info*, and *state-sync*. + +### Consensus/block execution methods + +The first time a new blockchain is started, CometBFT calls `InitChain`. From then on, method +`FinalizeBlock` is executed upon the decision of each block, resulting in an updated Application +state. During the execution of an instance of consensus, which decides the block for a given +height, and before method `FinalizeBlock` is called, methods `PrepareProposal`, `ProcessProposal`, +`ExtendVote`, and `VerifyVoteExtension` may be called several times. See +[CometBFT's expected behavior](./abci++_comet_expected_behavior.md) for details on the possible +call sequences of these methods. + +- [**InitChain:**](./abci++_methods.md#initchain) This method initializes the blockchain. + CometBFT calls it once upon genesis. + +- [**PrepareProposal:**](./abci++_methods.md#prepareproposal) It allows the block + proposer to perform application-dependent work in a block before proposing it. + This enables, for instance, batch optimizations to a block, which has been empirically + demonstrated to be a key component for improved performance. Method `PrepareProposal` is called + every time CometBFT is about to broadcast a Proposal message and *validValue* is `nil`. + CometBFT gathers outstanding transactions from the + mempool, generates a block header, and uses them to create a block to propose. Then, it calls + `RequestPrepareProposal` with the newly created proposal, called *raw proposal*. The Application + can make changes to the raw proposal, such as reordering, adding and removing transactions, before returning the + (potentially) modified proposal, called *prepared proposal* in the `ResponsePrepareProposal`. + The logic modifying the raw proposal MAY be non-deterministic. + +- [**ProcessProposal:**](./abci++_methods.md#processproposal) It allows a validator to + perform application-dependent work in a proposed block. This enables features such as immediate + block execution, and allows the Application to reject invalid blocks. + + CometBFT calls it when it receives a proposal and *validValue* is `nil`. + The Application cannot modify the proposal at this point but can reject it if + invalid. If that is the case, the consensus algorithm will prevote `nil` on the proposal, which has + strong liveness implications for CometBFT. As a general rule, the Application + SHOULD accept a prepared proposal passed via `ProcessProposal`, even if a part of + the proposal is invalid (e.g., an invalid transaction); the Application can + ignore the invalid part of the prepared proposal at block execution time. + The logic in `ProcessProposal` MUST be deterministic. + +- [**ExtendVote:**](./abci++_methods.md#extendvote) It allows applications to let their + validators do more than just validate within consensus. `ExtendVote` allows applications to + include non-deterministic data, opaque to the consensus algorithm, to precommit messages (the final round of + voting). The data, called *vote extension*, will be broadcast and received together with the + vote it is extending, and will be made available to the Application in the next height, + in the rounds where the local process is the proposer. + CometBFT calls `ExtendVote` when the consensus algorithm is about to send a non-`nil` precommit message. + If the Application does not have vote extension information to provide at that time, it returns + a 0-length byte array as its vote extension. + The logic in `ExtendVote` MAY be non-deterministic. + +- [**VerifyVoteExtension:**](./abci++_methods.md#verifyvoteextension) It allows + validators to validate the vote extension data attached to a precommit message. If the validation + fails, the whole precommit message will be deemed invalid and ignored by consensus algorithm. + This has a negative impact on liveness, i.e., if vote extensions repeatedly cannot be + verified by correct validators, the consensus algorithm may not be able to finalize a block even if sufficiently + many (+2/3) validators send precommit votes for that block. Thus, `VerifyVoteExtension` + should be implemented with special care. + As a general rule, an Application that detects an invalid vote extension SHOULD + accept it in `ResponseVerifyVoteExtension` and ignore it in its own logic. CometBFT calls it when + a process receives a precommit message with a (possibly empty) vote extension, for the current height. It is not called for precommit votes received after the height is concluded but while waiting to accumulate more precommit votes. + The logic in `VerifyVoteExtension` MUST be deterministic. + +- [**FinalizeBlock:**](./abci++_methods.md#finalizeblock) It delivers a decided block to the + Application. The Application must execute the transactions in the block deterministically and + update its state accordingly. Cryptographic commitments to the block and transaction results, + returned via the corresponding parameters in `ResponseFinalizeBlock`, are included in the header + of the next block. CometBFT calls it when a new block is decided. + When calling `FinalizeBlock` with a block, the consensus algorithm run by CometBFT guarantees + that at least one non-byzantine validator has run `ProcessProposal` on that block. + +- [**Commit:**](./abci++_methods.md#commit) Instructs the Application to persist its + state. It is a fundamental part of CometBFT's crash-recovery mechanism that ensures the + synchronization between CometBFT and the Application upon recovery. CometBFT calls it just after + having persisted the data returned by calls to `ResponseFinalizeBlock`. The Application can now discard + any state or data except the one resulting from executing the transactions in the decided block. + +### Mempool methods + +- [**CheckTx:**](./abci++_methods.md#checktx) This method allows the Application to validate + transactions. Validation can be stateless (e.g., checking signatures ) or stateful + (e.g., account balances). The type of validation performed is up to the application. If a + transaction passes the validation, then CometBFT adds it to the mempool; otherwise the + transaction is discarded. + CometBFT calls it when it receives a new transaction either coming from an external + user (e.g., a client) or another node. Furthermore, CometBFT can be configured to call + re-`CheckTx` on all outstanding transactions in the mempool after calling `Commit` for a block. + +### Info methods + +- [**Info:**](./abci++_methods.md#info) Used to sync CometBFT with the Application during a + handshake that happens upon recovery, or on startup when state-sync is used. + +- [**Query:**](./abci++_methods.md#query) This method can be used to query the Application for + information about the application state. + +### State-sync methods + +State sync allows new nodes to rapidly bootstrap by discovering, fetching, and applying +state machine (application) snapshots instead of replaying historical blocks. For more details, see the +[state sync documentation](../p2p/legacy-docs/messages/state-sync.md). + +New nodes discover and request snapshots from other nodes in the P2P network. +A CometBFT node that receives a request for snapshots from a peer will call +`ListSnapshots` on its Application. The Application returns the list of locally available +snapshots. +Note that the list does not contain the actual snapshots but metadata about them: height at which +the snapshot was taken, application-specific verification data and more (see +[snapshot data type](./abci++_methods.md#snapshot) for more details). After receiving a +list of available snapshots from a peer, the new node can offer any of the snapshots in the list to +its local Application via the `OfferSnapshot` method. The Application can check at this point the +validity of the snapshot metadata. + +Snapshots may be quite large and are thus broken into smaller "chunks" that can be +assembled into the whole snapshot. Once the Application accepts a snapshot and +begins restoring it, CometBFT will fetch snapshot "chunks" from existing nodes. +The node providing "chunks" will fetch them from its local Application using +the `LoadSnapshotChunk` method. + +As the new node receives "chunks" it will apply them sequentially to the local +application with `ApplySnapshotChunk`. When all chunks have been applied, the +Application's `AppHash` is retrieved via an `Info` query. +To ensure that the sync proceeded correctly, CometBFT compares the local Application's `AppHash` +to the `AppHash` stored on the blockchain (verified via +[light client verification](../light-client/verification/README.md)). + +In summary: + +- [**ListSnapshots:**](./abci++_methods.md#listsnapshots) Used by nodes to discover available + snapshots on peers. + +- [**OfferSnapshot:**](./abci++_methods.md#offersnapshot) When a node receives a snapshot from a + peer, CometBFT uses this method to offer the snapshot to the Application. + +- [**LoadSnapshotChunk:**](./abci++_methods.md#loadsnapshotchunk) Used by CometBFT to retrieve + snapshot chunks from the Application to send to peers. + +- [**ApplySnapshotChunk:**](./abci++_methods.md#applysnapshotchunk) Used by CometBFT to hand + snapshot chunks to the Application. + +### Other methods + +Additionally, there is a [**Flush**](./abci++_methods.md#flush) method that is called on every connection, +and an [**Echo**](./abci++_methods.md#echo) method that is used for debugging. + +More details on managing state across connections can be found in the section on +[Managing Application State](./abci%2B%2B_app_requirements.md#managing-the-application-state-and-related-topics). + +## Proposal timeout + +`PrepareProposal` stands on the consensus algorithm critical path, +i.e., CometBFT cannot make progress while this method is being executed. +Hence, if the Application takes a long time preparing a proposal, +the default value of *TimeoutPropose* might not be sufficient +to accommodate the method's execution and validator nodes might time out and prevote `nil`. +The proposal, in this case, will probably be rejected and a new round will be necessary. + +Timeouts are automatically increased for each new round of a height and, if the execution of `PrepareProposal` is bound, eventually *TimeoutPropose* will be long enough to accommodate the execution of `PrepareProposal`. +However, relying on this self adaptation could lead to performance degradation and, therefore, +operators are suggested to adjust the initial value of *TimeoutPropose* in CometBFT's configuration file, +in order to suit the needs of the particular application being deployed. + +This is particularly important if applications implement *immediate execution*. +To implement this technique, proposers need to execute the block being proposed within `PrepareProposal`, which could take longer than *TimeoutPropose*. + +## Deterministic State-Machine Replication + +[↑ Back to Outline](#outline) + +ABCI applications must implement deterministic finite-state machines to be +securely replicated by the CometBFT consensus engine. This means block execution +must be strictly deterministic: given the same +ordered set of transactions, all nodes will compute identical responses, for all +successive `FinalizeBlock` calls. This is critical because the +responses are included in the header of the next block, either via a Merkle root +or directly, so all nodes must agree on exactly what they are. + +For this reason, it is recommended that application state is not exposed to any +external user or process except via the ABCI connections to a consensus engine +like CometBFT. The Application must only change its state based on input +from block execution (`FinalizeBlock` calls), and not through +any other kind of request. This is the only way to ensure all nodes see the same +transactions and compute the same results. + +Applications that implement immediate execution (execute the blocks +that are about to be proposed, in `PrepareProposal`, or that require validation, in `ProcessProposal`) produce a new candidate state before a block is decided. +The state changes caused by processing those +proposed blocks must never replace the previous state until `FinalizeBlock` confirms +that the proposed block was decided and `Commit` is invoked for it. + +The same is true to Applications that quickly accept blocks and execute the +blocks optimistically in parallel with the remaining consensus steps to save +time during `FinalizeBlock`; they must only apply state changes in `Commit`. + +Additionally, vote extensions or the validation thereof (via `ExtendVote` or +`VerifyVoteExtension`) must *never* have side effects on the current state. +They can only be used when their data is provided in a `RequestPrepareProposal` call but, again, +without side effects to the app state. + +If there is some non-determinism in the state machine, consensus will eventually +fail as nodes disagree over the correct values for the block header. The +non-determinism must be fixed and the nodes restarted. + +Sources of non-determinism in applications may include: + +- Hardware failures + - Cosmic rays, overheating, etc. +- Node-dependent state + - Random numbers + - Time +- Underspecification + - Library version changes + - Race conditions + - Floating point numbers + - JSON or protobuf serialization + - Iterating through hash-tables/maps/dictionaries +- External Sources + - Filesystem + - Network calls (eg. some external REST API service) + +See [#56](https://github.com/tendermint/abci/issues/56) for the original discussion. + +Note that some methods (e.g., `Query` and `FinalizeBlock`) may return +non-deterministic data in the form of `Info`, `Log` and/or `Events` fields. The +`Log` is intended for the literal output from the Application's logger, while +the `Info` is any additional info that should be returned. These fields are not +included in block header computations, so we don't need agreement on them. See +each field's description on whether it must be deterministic or not. + +## Events + +[↑ Back to Outline](#outline) + +Method `FinalizeBlock` includes an `events` field at the top level in its +`Response*`, and one `events` field per transaction included in the block. +Applications may respond to this ABCI 2.0 method with an event list for each executed +transaction, and a general event list for the block itself. +Events allow applications to associate metadata with transactions and blocks. +Events returned via `FinalizeBlock` do not impact the consensus algorithm in any way +and instead exist to power subscriptions and queries of CometBFT state. + +An `Event` contains a `type` and a list of `EventAttributes`, which are key-value +string pairs denoting metadata about what happened during the method's (or transaction's) +execution. `Event` values can be used to index transactions and blocks according to what +happened during their execution. + +Each event has a `type` which is meant to categorize the event for a particular +`Response*` or `Tx`. A `Response*` or `Tx` may contain multiple events with duplicate +`type` values, where each distinct entry is meant to categorize attributes for a +particular event. Every key and value in an event's attributes must be UTF-8 +encoded strings along with the event type itself. + +```protobuf +message Event { + string type = 1; + repeated EventAttribute attributes = 2; +} +``` + +The attributes of an `Event` consist of a `key`, a `value`, and an `index` +flag. The index flag notifies the CometBFT indexer to index the attribute. + +The `type` and `attributes` fields are non-deterministic and may vary across +different nodes in the network. + +```protobuf +message EventAttribute { + string key = 1; + string value = 2; + bool index = 3; // nondeterministic +} +``` + +Example: + +```go + abci.ResponseFinalizeBlock{ + // ... + Events: []abci.Event{ + { + Type: "validator.provisions", + Attributes: []abci.EventAttribute{ + abci.EventAttribute{Key: "address", Value: "...", Index: true}, + abci.EventAttribute{Key: "amount", Value: "...", Index: true}, + abci.EventAttribute{Key: "balance", Value: "...", Index: true}, + }, + }, + { + Type: "validator.provisions", + Attributes: []abci.EventAttribute{ + abci.EventAttribute{Key: "address", Value: "...", Index: true}, + abci.EventAttribute{Key: "amount", Value: "...", Index: false}, + abci.EventAttribute{Key: "balance", Value: "...", Index: false}, + }, + }, + { + Type: "validator.slashed", + Attributes: []abci.EventAttribute{ + abci.EventAttribute{Key: "address", Value: "...", Index: false}, + abci.EventAttribute{Key: "amount", Value: "...", Index: true}, + abci.EventAttribute{Key: "reason", Value: "...", Index: true}, + }, + }, + // ... + }, +} +``` + +## Evidence + +[↑ Back to Outline](#outline) + +CometBFT's security model relies on the use of evidences of misbehavior. An evidence is an +irrefutable proof of malicious behavior by a network participant. It is the responsibility of +CometBFT to detect such malicious behavior. When malicious behavior is detected, CometBFT +will gossip evidences of misbehavior to other nodes and commit the evidences to +the chain once they are verified by a subset of validators. These evidences will then be +passed on to the Application through ABCI++. It is the responsibility of the +Application to handle evidence of misbehavior and exercise punishment. + +There are two forms of evidence: Duplicate Vote and Light Client Attack. More +information can be found in either [data structures](../core/data_structures.md) +or [accountability](../light-client/accountability/). + +EvidenceType has the following protobuf format: + +```protobuf +enum EvidenceType { + UNKNOWN = 0; + DUPLICATE_VOTE = 1; + LIGHT_CLIENT_ATTACK = 2; +} +``` + +## Errors + +[↑ Back to Outline](#outline) + +The `Query` and `CheckTx` methods include a `Code` field in their `Response*`. +Field `Code` is meant to contain an application-specific response code. +A response code of `0` indicates no error. Any other response code +indicates to CometBFT that an error occurred. + +These methods also return a `Codespace` string to CometBFT. This field is +used to disambiguate `Code` values returned by different domains of the +Application. The `Codespace` is a namespace for the `Code`. + +Methods `Echo`, `Info`, `Commit` and `InitChain` do not return errors. +An error in any of these methods represents a critical issue that CometBFT +has no reasonable way to handle. If there is an error in one +of these methods, the Application must crash to ensure that the error is safely +handled by an operator. + +Method `FinalizeBlock` is a special case. It contains a number of +`Code` and `Codespace` fields as part of type `ExecTxResult`. Each of +these codes reports errors related to the transaction it is attached to. +However, `FinalizeBlock` does not return errors at the top level, so the +same considerations on critical issues made for `Echo`, `Info`, and +`InitChain` also apply here. + +The handling of non-zero response codes by CometBFT is described below. + +### `CheckTx` + +When CometBFT receives a `ResponseCheckTx` with a non-zero `Code`, the associated +transaction will not be added to CometBFT's mempool or it will be removed if +it is already included. + +### `ExecTxResult` (as part of `FinalizeBlock`) + +The `ExecTxResult` type delivers transaction results from the Application to CometBFT. When +CometBFT receives a `ResponseFinalizeBlock` containing an `ExecTxResult` with a non-zero `Code`, +the response code is logged. Past `Code` values can be queried by clients. As the transaction was +part of a decided block, the `Code` does not influence consensus. + +### `Query` + +When CometBFT receives a `ResponseQuery` with a non-zero `Code`, this code is +returned directly to the client that initiated the query. diff --git a/cometbft/v0.38/spec/abci/Overview.mdx b/cometbft/v0.38/spec/abci/Overview.mdx new file mode 100644 index 00000000..fc83543f --- /dev/null +++ b/cometbft/v0.38/spec/abci/Overview.mdx @@ -0,0 +1,42 @@ +--- +order: 1 +parent: + title: ABCI++ + order: 3 +--- +{/* trigger rebuild */} + +# ABCI++ + +## Introduction + +ABCI++ is a major evolution of ABCI (**A**pplication **B**lock**c**hain **I**nterface). +Like its predecessor, ABCI++ is the interface between CometBFT (a state-machine +replication engine) and the actual state machine being replicated (i.e., the Application). +The API consists of a set of _methods_, each with a corresponding `Request` and `Response` +message type. + +The methods are always initiated by CometBFT. The Application implements its logic +for handling all ABCI++ methods. +Thus, CometBFT always sends the `Request*` messages and receives the `Response*` messages +in return. + +All ABCI++ messages and methods are defined in [protocol buffers](https://github.com/cometbft/cometbft/blob/v0.38.x/proto/tendermint/abci/types.proto). +This allows CometBFT to run with applications written in many programming languages. + +This specification is split as follows: + +- [Overview and basic concepts](/cometbft/v0.38/spec/abci/Outline) - interface's overview and concepts + needed to understand other parts of this specification. +- [Methods](/cometbft/v0.38/spec/abci/Methods) - complete details on all ABCI++ methods + and message types. +- [Requirements for the Application](/cometbft/v0.38/spec/abci/Requirements-for-the-Application) - formal requirements + on the Application's logic to ensure CometBFT properties such as liveness. These requirements define what + CometBFT expects from the Application; second part on managing ABCI application state and related topics. +- [CometBFT's expected behavior](/cometbft/v0.38/spec/abci/CometBFTs-expected-behavior) - specification of + how the different ABCI++ methods may be called by CometBFT. This explains what the Application + is to expect from CometBFT. +- [Example scenarios](/cometbft/v0.38/spec/abci/Introduction) - specific scenarios showing why the Application needs to account +for any CometBFT's behaviour prescribed by the specification. +- [Client and Server](/cometbft/v0.38/spec/abci/Client-and-server) - for those looking to implement their + own ABCI application servers. diff --git a/cometbft/v0.38/spec/abci/Requirements-for-the-Application.mdx b/cometbft/v0.38/spec/abci/Requirements-for-the-Application.mdx new file mode 100644 index 00000000..3c5c5e1c --- /dev/null +++ b/cometbft/v0.38/spec/abci/Requirements-for-the-Application.mdx @@ -0,0 +1,1070 @@ +--- +order: 3 +title: Requirements for the Application +--- + +# Requirements for the Application + +- [Requirements for the Application](#requirements-for-the-application) + - [Formal Requirements](#formal-requirements) + - [Consensus Connection Requirements](#consensus-connection-requirements) + - [Mempool Connection Requirements](#mempool-connection-requirements) + - [Managing the Application state and related topics](#managing-the-application-state-and-related-topics) + - [Connection State](#connection-state) + - [Concurrency](#concurrency) + - [FinalizeBlock](#finalizeblock) + - [Commit](#commit) + - [Candidate States](#candidate-states) + - [States and ABCI++ Connections](#states-and-abci-connections) + - [Consensus Connection](#consensus-connection) + - [Mempool Connection](#mempool-connection) + - [Replay Protection](#replay-protection) + - [Info/Query Connection](#infoquery-connection) + - [Snapshot Connection](#snapshot-connection) + - [Transaction Results](#transaction-results) + - [Gas](#gas) + - [Specifics of `ResponseCheckTx`](#specifics-of-responsechecktx) + - [Specifics of `ExecTxResult`](#specifics-of-exectxresult) + - [Updating the Validator Set](#updating-the-validator-set) + - [Consensus Parameters](#consensus-parameters) + - [List of Parameters](#list-of-parameters) + - [ABCIParams.VoteExtensionsEnableHeight](#abciparamsvoteextensionsenableheight) + - [BlockParams.MaxBytes](#blockparamsmaxbytes) + - [BlockParams.MaxGas](#blockparamsmaxgas) + - [EvidenceParams.MaxAgeDuration](#evidenceparamsmaxageduration) + - [EvidenceParams.MaxAgeNumBlocks](#evidenceparamsmaxagenumblocks) + - [EvidenceParams.MaxBytes](#evidenceparamsmaxbytes) + - [ValidatorParams.PubKeyTypes](#validatorparamspubkeytypes) + - [VersionParams.App](#versionparamsapp) + - [Updating Consensus Parameters](#updating-consensus-parameters) + - [`InitChain`](#initchain) + - [`FinalizeBlock`, `PrepareProposal`/`ProcessProposal`](#finalizeblock-prepareproposalprocessproposal) + - [`Query`](#query) + - [Query Proofs](#query-proofs) + - [Peer Filtering](#peer-filtering) + - [Paths](#paths) + - [Crash Recovery](#crash-recovery) + - [State Sync](#state-sync) + - [Taking Snapshots](#taking-snapshots) + - [Bootstrapping a Node](#bootstrapping-a-node) + - [Snapshot Discovery](#snapshot-discovery) + - [Snapshot Restoration](#snapshot-restoration) + - [Snapshot Verification](#snapshot-verification) + - [Transition to Consensus](#transition-to-consensus) + - [Application configuration required to switch to ABCI 2.0](#application-configuration-required-to-switch-to-abci-20) + + +## Formal Requirements + +### Consensus Connection Requirements + +This section specifies what CometBFT expects from the Application. It is structured as a set +of formal requirements that can be used for testing and verification of the Application's logic. + +Let *p* and *q* be two correct processes. +Let *rp* (resp. *rq*) be a round of height *h* where *p* (resp. *q*) is the +proposer. +Let *sp,h-1* be *p*'s Application's state committed for height *h-1*. +Let *vp* (resp. *vq*) be the block that *p*'s (resp. *q*'s) CometBFT passes +on to the Application +via `RequestPrepareProposal` as proposer of round *rp* (resp *rq*), height *h*, +also known as the raw proposal. +Let *up* (resp. *uq*) the possibly modified block *p*'s (resp. *q*'s) Application +returns via `ResponsePrepareProposal` to CometBFT, also known as the prepared proposal. + +Process *p*'s prepared proposal can differ in two different rounds where *p* is the proposer. + +- Requirement 1 [`PrepareProposal`, timeliness]: If *p*'s Application fully executes prepared blocks in + `PrepareProposal` and the network is in a synchronous period while processes *p* and *q* are in *rp*, + then the value of *TimeoutPropose* at *q* must be such that *q*'s propose timer does not time out + (which would result in *q* prevoting `nil` in *rp*). + +Full execution of blocks at `PrepareProposal` time stands on CometBFT's critical path. Thus, +Requirement 1 ensures the Application or operator will set a value for `TimeoutPropose` such that the time it takes +to fully execute blocks in `PrepareProposal` does not interfere with CometBFT's propose timer. +Note that the violation of Requirement 1 may lead to further rounds, but will not +compromise liveness because even though `TimeoutPropose` is used as the initial +value for proposal timeouts, CometBFT will be dynamically adjust these timeouts +such that they will eventually be enough for completing `PrepareProposal`. + +- Requirement 2 [`PrepareProposal`, tx-size]: When *p*'s Application calls `ResponsePrepareProposal`, the + total size in bytes of the transactions returned does not exceed `RequestPrepareProposal.max_tx_bytes`. + +Busy blockchains might seek to gain full visibility into transactions in CometBFT's mempool, +rather than having visibility only on *a* subset of those transactions that fit in a block. +The application can do so by setting `ConsensusParams.Block.MaxBytes` to -1. +This instructs CometBFT (a) to enforce the maximum possible value for `MaxBytes` (100 MB) at CometBFT level, +and (b) to provide *all* transactions in the mempool when calling `RequestPrepareProposal`. +Under these settings, the aggregated size of all transactions may exceed `RequestPrepareProposal.max_tx_bytes`. +Hence, Requirement 2 ensures that the size in bytes of the transaction list returned by the application will never +cause the resulting block to go beyond its byte size limit. + +- Requirement 3 [`PrepareProposal`, `ProcessProposal`, coherence]: For any two correct processes *p* and *q*, + if *q*'s CometBFT calls `RequestProcessProposal` on *up*, + *q*'s Application returns Accept in `ResponseProcessProposal`. + +Requirement 3 makes sure that blocks proposed by correct processes *always* pass the correct receiving process's +`ProcessProposal` check. +On the other hand, if there is a deterministic bug in `PrepareProposal` or `ProcessProposal` (or in both), +strictly speaking, this makes all processes that hit the bug byzantine. This is a problem in practice, +as very often validators are running the Application from the same codebase, so potentially *all* would +likely hit the bug at the same time. This would result in most (or all) processes prevoting `nil`, with the +serious consequences on CometBFT's liveness that this entails. Due to its criticality, Requirement 3 is a +target for extensive testing and automated verification. + +- Requirement 4 [`ProcessProposal`, determinism-1]: `ProcessProposal` is a (deterministic) function of the current + state and the block that is about to be applied. In other words, for any correct process *p*, and any arbitrary block *u*, + if *p*'s CometBFT calls `RequestProcessProposal` on *u* at height *h*, + then *p*'s Application's acceptance or rejection **exclusively** depends on *u* and *sp,h-1*. + +- Requirement 5 [`ProcessProposal`, determinism-2]: For any two correct processes *p* and *q*, and any arbitrary + block *u*, + if *p*'s (resp. *q*'s) CometBFT calls `RequestProcessProposal` on *u* at height *h*, + then *p*'s Application accepts *u* if and only if *q*'s Application accepts *u*. + Note that this requirement follows from Requirement 4 and the Agreement property of consensus. + +Requirements 4 and 5 ensure that all correct processes will react in the same way to a proposed block, even +if the proposer is Byzantine. However, `ProcessProposal` may contain a bug that renders the +acceptance or rejection of the block non-deterministic, and therefore prevents processes hitting +the bug from fulfilling Requirements 4 or 5 (effectively making those processes Byzantine). +In such a scenario, CometBFT's liveness cannot be guaranteed. +Again, this is a problem in practice if most validators are running the same software, as they are likely +to hit the bug at the same point. There is currently no clear solution to help with this situation, so +the Application designers/implementors must proceed very carefully with the logic/implementation +of `ProcessProposal`. As a general rule `ProcessProposal` SHOULD always accept the block. + +According to the Tendermint consensus algorithm, currently adopted in CometBFT, +a correct process can broadcast at most one precommit +message in round *r*, height *h*. +Since, as stated in the [Methods](./abci++_methods.md#extendvote) section, `ResponseExtendVote` +is only called when the consensus algorithm +is about to broadcast a non-`nil` precommit message, a correct process can only produce one vote extension +in round *r*, height *h*. +Let *erp* be the vote extension that the Application of a correct process *p* returns via +`ResponseExtendVote` in round *r*, height *h*. +Let *wrp* be the proposed block that *p*'s CometBFT passes to the Application via `RequestExtendVote` +in round *r*, height *h*. + +- Requirement 6 [`ExtendVote`, `VerifyVoteExtension`, coherence]: For any two different correct + processes *p* and *q*, if *q* receives *erp* from *p* in height *h*, *q*'s + Application returns Accept in `ResponseVerifyVoteExtension`. + +Requirement 6 constrains the creation and handling of vote extensions in a similar way as Requirement 3 +constrains the creation and handling of proposed blocks. +Requirement 6 ensures that extensions created by correct processes *always* pass the `VerifyVoteExtension` +checks performed by correct processes receiving those extensions. +However, if there is a (deterministic) bug in `ExtendVote` or `VerifyVoteExtension` (or in both), +we will face the same liveness issues as described for Requirement 5, as Precommit messages with invalid vote +extensions will be discarded. + +- Requirement 7 [`VerifyVoteExtension`, determinism-1]: `VerifyVoteExtension` is a (deterministic) function of + the current state, the vote extension received, and the prepared proposal that the extension refers to. + In other words, for any correct process *p*, and any arbitrary vote extension *e*, and any arbitrary + block *w*, if *p*'s (resp. *q*'s) CometBFT calls `RequestVerifyVoteExtension` on *e* and *w* at height *h*, + then *p*'s Application's acceptance or rejection **exclusively** depends on *e*, *w* and *sp,h-1*. + +- Requirement 8 [`VerifyVoteExtension`, determinism-2]: For any two correct processes *p* and *q*, + and any arbitrary vote extension *e*, and any arbitrary block *w*, + if *p*'s (resp. *q*'s) CometBFT calls `RequestVerifyVoteExtension` on *e* and *w* at height *h*, + then *p*'s Application accepts *e* if and only if *q*'s Application accepts *e*. + Note that this requirement follows from Requirement 7 and the Agreement property of consensus. + +Requirements 7 and 8 ensure that the validation of vote extensions will be deterministic at all +correct processes. +Requirements 7 and 8 protect against arbitrary vote extension data from Byzantine processes, +in a similar way as Requirements 4 and 5 protect against arbitrary proposed blocks. +Requirements 7 and 8 can be violated by a bug inducing non-determinism in +`VerifyVoteExtension`. In this case liveness can be compromised. +Extra care should be put in the implementation of `ExtendVote` and `VerifyVoteExtension`. +As a general rule, `VerifyVoteExtension` SHOULD always accept the vote extension. + +- Requirement 9 [*all*, no-side-effects]: *p*'s calls to `RequestPrepareProposal`, + `RequestProcessProposal`, `RequestExtendVote`, and `RequestVerifyVoteExtension` at height *h* do + not modify *sp,h-1*. + + +- Requirement 10 [`ExtendVote`, `FinalizeBlock`, non-dependency]: for any correct process *p*, +and any vote extension *e* that *p* received at height *h*, the computation of +*sp,h* does not depend on *e*. + +The call to correct process *p*'s `RequestFinalizeBlock` at height *h*, with block *vp,h* +passed as parameter, creates state *sp,h*. +Additionally, *p*'s `FinalizeBlock` creates a set of transaction results *Tp,h*. + +- Requirement 11 [`FinalizeBlock`, determinism-1]: For any correct process *p*, + *sp,h* exclusively depends on *sp,h-1* and *vp,h*. + +- Requirement 12 [`FinalizeBlock`, determinism-2]: For any correct process *p*, + the contents of *Tp,h* exclusively depend on *sp,h-1* and *vp,h*. + +Note that Requirements 11 and 12, combined with the Agreement property of consensus ensure +state machine replication, i.e., the Application state evolves consistently at all correct processes. + +Also, notice that neither `PrepareProposal` nor `ExtendVote` have determinism-related +requirements associated. +Indeed, `PrepareProposal` is not required to be deterministic: + +- *up* may depend on *vp* and *sp,h-1*, but may also depend on other values or operations. +- *vp = vq ⇏ up = uq*. + +Likewise, `ExtendVote` can also be non-deterministic: + +- *erp* may depend on *wrp* and *sp,h-1*, + but may also depend on other values or operations. +- *wrp = wrq ⇏ + erp = erq* + +### Mempool Connection Requirements + +Let *CheckTxCodestx,p,h* denote the set of result codes returned by *p*'s Application, +via `ResponseCheckTx`, +to successive calls to `RequestCheckTx` occurring while the Application is at height *h* +and having transaction *tx* as parameter. +*CheckTxCodestx,p,h* is a set since *p*'s Application may +return different result codes during height *h*. +If *CheckTxCodestx,p,h* is a singleton set, i.e. the Application always returned +the same result code in `ResponseCheckTx` while at height *h*, +we define *CheckTxCodetx,p,h* as the singleton value of *CheckTxCodestx,p,h*. +If *CheckTxCodestx,p,h* is not a singleton set, *CheckTxCodetx,p,h* is undefined. +Let predicate *OK(CheckTxCodetx,p,h)* denote whether *CheckTxCodetx,p,h* is `SUCCESS`. + +- Requirement 13 [`CheckTx`, eventual non-oscillation]: For any transaction *tx*, + there exists a boolean value *b*, + and a height *hstable* such that, + for any correct process *p*, + *CheckTxCodetx,p,h* is defined, and + *OK(CheckTxCodetx,p,h) = b* + for any height *h ≥ hstable*. + +Requirement 13 ensures that +a transaction will eventually stop oscillating between `CheckTx` success and failure +if it stays in *p's* mempool for long enough. +This condition on the Application's behavior allows the mempool to ensure that +a transaction will leave the mempool of all full nodes, +either because it is expunged everywhere due to failing `CheckTx` calls, +or because it stays valid long enough to be gossipped, proposed and decided. +Although Requirement 13 defines a global *hstable*, application developers +can consider such stabilization height as local to process *p* (*hp,stable*), +without loss for generality. +In contrast, the value of *b* MUST be the same across all processes. + +## Managing the Application state and related topics + +### Connection State + +CometBFT maintains four concurrent ABCI++ connections, namely +[Consensus Connection](#consensus-connection), +[Mempool Connection](#mempool-connection), +[Info/Query Connection](#infoquery-connection), and +[Snapshot Connection](#snapshot-connection). +It is common for an application to maintain a distinct copy of +the state for each connection, which are synchronized upon `Commit` calls. + +#### Concurrency + +In principle, each of the four ABCI++ connections operates concurrently with one +another. This means applications need to ensure access to state is +thread safe. Both the +[default in-process ABCI client](https://github.com/cometbft/cometbft/blob/v0.38.x/abci/client/local_client.go#L13) +and the +[default Go ABCI server](https://github.com/cometbft/cometbft/blob/v0.38.x/abci/server/socket_server.go#L20) +use a global lock to guard the handling of events across all connections, so they are not +concurrent at all. This means whether your app is compiled in-process with +CometBFT using the `NewLocalClient`, or run out-of-process using the `SocketServer`, +ABCI messages from all connections are received in sequence, one at a +time. + +The existence of this global mutex means Go application developers can get thread safety for application state by routing all reads and writes through the ABCI system. Thus it may be unsafe to expose application state directly to an RPC interface, and unless explicit measures are taken, all queries should be routed through the ABCI Query method. + +#### FinalizeBlock + +When the consensus algorithm decides on a block, CometBFT uses `FinalizeBlock` to send the +decided block's data to the Application, which uses it to transition its state, but MUST NOT persist it; +persisting MUST be done during `Commit`. + +The Application must remember the latest height from which it +has run a successful `Commit` so that it can tell CometBFT where to +pick up from when it recovers from a crash. See information on the Handshake +[here](#crash-recovery). + +#### Commit + +The Application should persist its state during `Commit`, before returning from it. + +Before invoking `Commit`, CometBFT locks the mempool and flushes the mempool connection. This ensures that +no new messages +will be received on the mempool connection during this processing step, providing an opportunity to safely +update all four +connection states to the latest committed state at the same time. + +When `Commit` returns, CometBFT unlocks the mempool. + +WARNING: if the ABCI app logic processing the `Commit` message sends a +`/broadcast_tx_sync` or `/broadcast_tx` and waits for the response +before proceeding, it will deadlock. Executing `broadcast_tx` calls +involves acquiring the mempool lock that CometBFT holds during the `Commit` call. +Synchronous mempool-related calls must be avoided as part of the sequential logic of the +`Commit` function. + +#### Candidate States + +CometBFT calls `PrepareProposal` when it is about to send a proposed block to the network. +Likewise, CometBFT calls `ProcessProposal` upon reception of a proposed block from the +network. The proposed block's data +that is disclosed to the Application by these two methods is the following: + +- the transaction list +- the `LastCommit` referring to the previous block +- the block header's hash (except in `PrepareProposal`, where it is not known yet) +- list of validators that misbehaved +- the block's timestamp +- `NextValidatorsHash` +- Proposer address + +The Application may decide to *immediately* execute the given block (i.e., upon `PrepareProposal` +or `ProcessProposal`). There are two main reasons why the Application may want to do this: + +- *Avoiding invalid transactions in blocks*. + In order to be sure that the block does not contain *any* invalid transaction, there may be + no way other than fully executing the transactions in the block as though it was the *decided* + block. +- *Quick `FinalizeBlock` execution*. + Upon reception of the decided block via `FinalizeBlock`, if that same block was executed + upon `PrepareProposal` or `ProcessProposal` and the resulting state was kept in memory, the + Application can simply apply that state (faster) to the main state, rather than reexecuting + the decided block (slower). + +`PrepareProposal`/`ProcessProposal` can be called many times for a given height. Moreover, +it is not possible to accurately predict which of the blocks proposed in a height will be decided, +being delivered to the Application in that height's `FinalizeBlock`. +Therefore, the state resulting from executing a proposed block, denoted a *candidate state*, should +be kept in memory as a possible final state for that height. When `FinalizeBlock` is called, the Application should +check if the decided block corresponds to one of its candidate states; if so, it will apply it as +its *ExecuteTxState* (see [Consensus Connection](#consensus-connection) below), +which will be persisted during the upcoming `Commit` call. + +Under adverse conditions (e.g., network instability), the consensus algorithm might take many rounds. +In this case, potentially many proposed blocks will be disclosed to the Application for a given height. +By the nature of Tendermint consensus algorithm, currently adopted in CometBFT, the number of proposed blocks received by the Application +for a particular height cannot be bound, so Application developers must act with care and use mechanisms +to bound memory usage. As a general rule, the Application should be ready to discard candidate states +before `FinalizeBlock`, even if one of them might end up corresponding to the +decided block and thus have to be reexecuted upon `FinalizeBlock`. + +### [States and ABCI++ Connections](#states-and-abci-connections) + +#### Consensus Connection + +The Consensus Connection should maintain an *ExecuteTxState* — the working state +for block execution. It should be updated by the call to `FinalizeBlock` +during block execution and committed to disk as the "latest +committed state" during `Commit`. Execution of a proposed block (via `PrepareProposal`/`ProcessProposal`) +**must not** update the *ExecuteTxState*, but rather be kept as a separate candidate state until `FinalizeBlock` +confirms which of the candidate states (if any) can be used to update *ExecuteTxState*. + +#### Mempool Connection + +The mempool Connection maintains *CheckTxState*. CometBFT sequentially processes an incoming +transaction (via RPC from client or P2P from the gossip layer) against *CheckTxState*. +If the processing does not return any error, the transaction is accepted into the mempool +and CometBFT starts gossipping it. +*CheckTxState* should be reset to the latest committed state +at the end of every `Commit`. + +During the execution of a consensus instance, the *CheckTxState* may be updated concurrently with the +*ExecuteTxState*, as messages may be sent concurrently on the Consensus and Mempool connections. +At the end of the consensus instance, as described above, CometBFT locks the mempool and flushes +the mempool connection before calling `Commit`. This ensures that all pending `CheckTx` calls are +responded to and no new ones can begin. + +After the `Commit` call returns, while still holding the mempool lock, `CheckTx` is run again on all +transactions that remain in the node's local mempool after filtering those included in the block. +Parameter `Type` in `RequestCheckTx` +indicates whether an incoming transaction is new (`CheckTxType_New`), or a +recheck (`CheckTxType_Recheck`). + +Finally, after re-checking transactions in the mempool, CometBFT will unlock +the mempool connection. New transactions are once again able to be processed through `CheckTx`. + +Note that `CheckTx` is just a weak filter to keep invalid transactions out of the mempool and, +ultimately, ouf of the blockchain. +Since the transaction cannot be guaranteed to be checked against the exact same state as it +will be executed as part of a (potential) decided block, `CheckTx` shouldn't check *everything* +that affects the transaction's validity, in particular those checks whose validity may depend on +transaction ordering. `CheckTx` is weak because a Byzantine node need not care about `CheckTx`; +it can propose a block full of invalid transactions if it wants. The mechanism ABCI++ has +in place for dealing with such behavior is `ProcessProposal`. + +##### Replay Protection + +It is possible for old transactions to be sent again to the Application. This is typically +undesirable for all transactions, except for a generally small subset of them which are idempotent. + +The mempool has a mechanism to prevent duplicated transactions from being processed. +This mechanism is nevertheless best-effort (currently based on the indexer) +and does not provide any guarantee of non duplication. +It is thus up to the Application to implement an application-specific +replay protection mechanism with strong guarantees as part of the logic in `CheckTx`. + +#### Info/Query Connection + +The Info (or Query) Connection should maintain a `QueryState`. This connection has two +purposes: 1) having the application answer the queries CometBFT receives from users +(see section [Query](#query)), +and 2) synchronizing CometBFT and the Application at start up time (see +[Crash Recovery](#crash-recovery)) +or after state sync (see [State Sync](#state-sync)). + +`QueryState` is a read-only copy of *ExecuteTxState* as it was after the last +`Commit`, i.e. +after the full block has been processed and the state committed to disk. + +#### Snapshot Connection + +The Snapshot Connection is used to serve state sync snapshots for other nodes +and/or restore state sync snapshots to a local node being bootstrapped. +Snapshot management is optional: an Application may choose not to implement it. + +For more information, see Section [State Sync](#state-sync). + +### Transaction Results + +The Application is expected to return a list of +[`ExecTxResult`](./abci%2B%2B_methods.md#exectxresult) in +[`ResponseFinalizeBlock`](./abci%2B%2B_methods.md#finalizeblock). The list of transaction +results MUST respect the same order as the list of transactions delivered via +[`RequestFinalizeBlock`](./abci%2B%2B_methods.md#finalizeblock). +This section discusses the fields inside this structure, along with the fields in +[`ResponseCheckTx`](./abci%2B%2B_methods.md#checktx), +whose semantics are similar. + +The `Info` and `Log` fields are +non-deterministic values for debugging/convenience purposes. CometBFT logs them but they +are otherwise ignored. + +#### Gas + +Ethereum introduced the notion of *gas* as an abstract representation of the +cost of the resources consumed by nodes when processing a transaction. Every operation in the +Ethereum Virtual Machine uses some amount of gas. +Gas has a market-variable price based on which miners can accept or reject to execute a +particular operation. + +Users propose a maximum amount of gas for their transaction; if the transaction uses less, they get +the difference credited back. CometBFT adopts a similar abstraction, +though uses it only optionally and weakly, allowing applications to define +their own sense of the cost of execution. + +In CometBFT, the [ConsensusParams.Block.MaxGas](#consensus-parameters) limits the amount of +total gas that can be used by all transactions in a block. +The default value is `-1`, which means the block gas limit is not enforced, or that the concept of +gas is meaningless. + +Responses contain a `GasWanted` and `GasUsed` field. The former is the maximum +amount of gas the sender of a transaction is willing to use, and the latter is how much it actually +used. Applications should enforce that `GasUsed <= GasWanted` — i.e. transaction execution +or validation should fail before it can use more resources than it requested. + +When `MaxGas > -1`, CometBFT enforces the following rules: + +- `GasWanted <= MaxGas` for every transaction in the mempool +- `(sum of GasWanted in a block) <= MaxGas` when proposing a block + +If `MaxGas == -1`, no rules about gas are enforced. + +In v0.34.x and earlier versions, CometBFT does not enforce anything about Gas in consensus, +only in the mempool. +This means it does not guarantee that committed blocks satisfy these rules. +It is the application's responsibility to return non-zero response codes when gas limits are exceeded +when executing the transactions of a block. +Since the introduction of `PrepareProposal` and `ProcessProposal` in v.0.37.x, it is now possible +for the Application to enforce that all blocks proposed (and voted for) in consensus — and thus all +blocks decided — respect the `MaxGas` limits described above. + +Since the Application should enforce that `GasUsed <= GasWanted` when executing a transaction, and +it can use `PrepareProposal` and `ProcessProposal` to enforce that `(sum of GasWanted in a block) <= MaxGas` +in all proposed or prevoted blocks, +we have: + +- `(sum of GasUsed in a block) <= MaxGas` for every block + +The `GasUsed` field is ignored by CometBFT. + +#### Specifics of `ResponseCheckTx` + +If `Code != 0`, it will be rejected from the mempool and hence +not broadcasted to other peers and not included in a proposal block. + +`Data` contains the result of the `CheckTx` transaction execution, if any. It does not need to be +deterministic since, given a transaction, nodes' Applications +might have a different *CheckTxState* values when they receive it and check their validity +via `CheckTx`. +CometBFT ignores this value in `ResponseCheckTx`. + +From v0.34.x on, there is a `Priority` field in `ResponseCheckTx` that can be +used to explicitly prioritize transactions in the mempool for inclusion in a block +proposal. + +#### Specifics of `ExecTxResult` + +`FinalizeBlock` is the workhorse of the blockchain. CometBFT delivers the decided block, +including the list of all its transactions synchronously to the Application. +The block delivered (and thus the transaction order) is the same at all correct nodes as guaranteed +by the Agreement property of consensus. + +The `Data` field in `ExecTxResult` contains an array of bytes with the transaction result. +It must be deterministic (i.e., the same value must be returned at all nodes), but it can contain arbitrary +data. Likewise, the value of `Code` must be deterministic. +If `Code != 0`, the transaction will be marked invalid, +though it is still included in the block. Invalid transactions are not indexed, as they are +considered analogous to those that failed `CheckTx`. + +Both the `Code` and `Data` are included in a structure that is hashed into the +`LastResultsHash` of the block header in the next height. + +`Events` include any events for the execution, which CometBFT will use to index +the transaction by. This allows transactions to be queried according to what +events took place during their execution. + +### Updating the Validator Set + +The application may set the validator set during +[`InitChain`](./abci%2B%2B_methods.md#initchain), and may update it during +[`FinalizeBlock`](./abci%2B%2B_methods.md#finalizeblock). In both cases, a structure of type +[`ValidatorUpdate`](./abci%2B%2B_methods.md#validatorupdate) is returned. + +The `InitChain` method, used to initialize the Application, can return a list of validators. +If the list is empty, CometBFT will use the validators loaded from the genesis +file. +If the list returned by `InitChain` is not empty, CometBFT will use its contents as the validator set. +This way the application can set the initial validator set for the +blockchain. + +Applications must ensure that a single set of validator updates does not contain duplicates, i.e. +a given public key can only appear once within a given update. If an update includes +duplicates, the block execution will fail irrecoverably. + +Structure `ValidatorUpdate` contains a public key, which is used to identify the validator: +The public key currently supports three types: + +- `ed25519` +- `secp256k1` +- `sr25519` + +Structure `ValidatorUpdate` also contains an `ìnt64` field denoting the validator's new power. +Applications must ensure that +`ValidatorUpdate` structures abide by the following rules: + +- power must be non-negative +- if power is set to 0, the validator must be in the validator set; it will be removed from the set +- if power is greater than 0: + - if the validator is not in the validator set, it will be added to the + set with the given power + - if the validator is in the validator set, its power will be adjusted to the given power +- the total power of the new validator set must not exceed `MaxTotalVotingPower`, where + `MaxTotalVotingPower = MaxInt64 / 8` + +Note the updates returned after processing the block at height `H` will only take effect +at block `H+2` (see Section [Methods](./abci%2B%2B_methods.md)). + +### Consensus Parameters + +`ConsensusParams` are global parameters that apply to all validators in a blockchain. +They enforce certain limits in the blockchain, like the maximum size +of blocks, amount of gas used in a block, and the maximum acceptable age of +evidence. They can be set in +[`InitChain`](./abci%2B%2B_methods.md#initchain), and updated in +[`FinalizeBlock`](./abci%2B%2B_methods.md#finalizeblock). +These parameters are deterministically set and/or updated by the Application, so +all full nodes have the same value at a given height. + +#### List of Parameters + +These are the current consensus parameters (as of v0.38.x): + +1. [ABCIParams.VoteExtensionsEnableHeight](#abciparamsvoteextensionsenableheight) +2. [BlockParams.MaxBytes](#blockparamsmaxbytes) +3. [BlockParams.MaxGas](#blockparamsmaxgas) +4. [EvidenceParams.MaxAgeDuration](#evidenceparamsmaxageduration) +5. [EvidenceParams.MaxAgeNumBlocks](#evidenceparamsmaxagenumblocks) +6. [EvidenceParams.MaxBytes](#evidenceparamsmaxbytes) +7. [ValidatorParams.PubKeyTypes](#validatorparamspubkeytypes) +8. [VersionParams.App](#versionparamsapp) + +##### ABCIParams.VoteExtensionsEnableHeight + +This parameter is either 0 or a positive height at which vote extensions +become mandatory. If the value is zero (which is the default), vote +extensions are not expected. Otherwise, at all heights greater than the +configured height `H` vote extensions must be present (even if empty). +When the configured height `H` is reached, `PrepareProposal` will not +include vote extensions yet, but `ExtendVote` and `VerifyVoteExtension` will +be called. Then, when reaching height `H+1`, `PrepareProposal` will +include the vote extensions from height `H`. For all heights after `H` + +- vote extensions cannot be disabled, +- they are mandatory: all precommit messages sent MUST have an extension + attached. Nevertheless, the application MAY provide 0-length + extensions. + +Must always be set to a future height, 0, or the same height that was previously set. +Once the chain's height reaches the value set, it cannot be changed to a different value. + +##### BlockParams.MaxBytes + +The maximum size of a complete Protobuf encoded block. +This is enforced by the consensus algorithm. + +This implies a maximum transaction size that is `MaxBytes`, less the expected size of +the header, the validator set, and any included evidence in the block. + +The Application should be aware that honest validators *may* produce and +broadcast blocks with up to the configured `MaxBytes` size. +As a result, the consensus +[timeout parameters](../../docs/core/configuration.md#consensus-timeouts-explained) +adopted by nodes should be configured so as to account for the worst-case +latency for the delivery of a full block with `MaxBytes` size to all validators. + +If the Application wants full control over the size of blocks, +it can do so by enforcing a byte limit set up at the Application level. +This Application-internal limit is used by `PrepareProposal` to bound the total size +of transactions it returns, and by `ProcessProposal` to reject any received block +whose total transaction size is bigger than the enforced limit. +In such case, the Application MAY set `MaxBytes` to -1. + +If the Application sets value -1, consensus will: + +- consider that the actual value to enforce is 100 MB +- will provide *all* transactions in the mempool in calls to `PrepareProposal` + +Must have `MaxBytes == -1` OR `0 < MaxBytes <= 100 MB`. + +> Bear in mind that the default value for the `BlockParams.MaxBytes` consensus +> parameter accepts as valid blocks with size up to 21 MB. +> If the Application's use case does not need blocks of that size, +> or if the impact (specially on bandwidth consumption and block latency) +> of propagating blocks of that size was not evaluated, +> it is strongly recommended to wind down this default value. + +##### BlockParams.MaxGas + +The maximum of the sum of `GasWanted` that will be allowed in a proposed block. +This is *not* enforced by the consensus algorithm. +It is left to the Application to enforce (ie. if transactions are included past the +limit, they should return non-zero codes). It is used by CometBFT to limit the +transactions included in a proposed block. + +Must have `MaxGas >= -1`. +If `MaxGas == -1`, no limit is enforced. + +##### EvidenceParams.MaxAgeDuration + +This is the maximum age of evidence in time units. +This is enforced by the consensus algorithm. + +If a block includes evidence older than this (AND the evidence was created more +than `MaxAgeNumBlocks` ago), the block will be rejected (validators won't vote +for it). + +Must have `MaxAgeDuration > 0`. + +##### EvidenceParams.MaxAgeNumBlocks + +This is the maximum age of evidence in blocks. +This is enforced by the consensus algorithm. + +If a block includes evidence older than this (AND the evidence was created more +than `MaxAgeDuration` ago), the block will be rejected (validators won't vote +for it). + +Must have `MaxAgeNumBlocks > 0`. + +##### EvidenceParams.MaxBytes + +This is the maximum size of total evidence in bytes that can be committed to a +single block. It should fall comfortably under the max block bytes. + +Its value must not exceed the size of +a block minus its overhead ( ~ `BlockParams.MaxBytes`). + +Must have `MaxBytes > 0`. + +##### ValidatorParams.PubKeyTypes + +The parameter restricts the type of keys validators can use. The parameter uses ABCI pubkey naming, not Amino names. + +##### VersionParams.App + +This is the version of the ABCI application. + +#### Updating Consensus Parameters + +The application may set the `ConsensusParams` during +[`InitChain`](./abci%2B%2B_methods.md#initchain), +and update them during +[`FinalizeBlock`](./abci%2B%2B_methods.md#finalizeblock). +If the `ConsensusParams` is empty, it will be ignored. Each field +that is not empty will be applied in full. For instance, if updating the +`Block.MaxBytes`, applications must also set the other `Block` fields (like +`Block.MaxGas`), even if they are unchanged, as they will otherwise cause the +value to be updated to the default. + +##### `InitChain` + +`ResponseInitChain` includes a `ConsensusParams` parameter. +If `ConsensusParams` is `nil`, CometBFT will use the params loaded in the genesis +file. If `ConsensusParams` is not `nil`, CometBFT will use it. +This way the application can determine the initial consensus parameters for the +blockchain. + +##### `FinalizeBlock`, `PrepareProposal`/`ProcessProposal` + +`ResponseFinalizeBlock` accepts a `ConsensusParams` parameter. +If `ConsensusParams` is `nil`, CometBFT will do nothing. +If `ConsensusParams` is not `nil`, CometBFT will use it. +This way the application can update the consensus parameters over time. + +The updates returned in block `H` will take effect right away for block +`H+1`. + +### `Query` + +`Query` is a generic method with lots of flexibility to enable diverse sets +of queries on application state. CometBFT makes use of `Query` to filter new peers +based on ID and IP, and exposes `Query` to the user over RPC. + +Note that calls to `Query` are not replicated across nodes, but rather query the +local node's state - hence they may return stale reads. For reads that require +consensus, use a transaction. + +The most important use of `Query` is to return Merkle proofs of the application state at some height +that can be used for efficient application-specific light-clients. + +Note CometBFT has technically no requirements from the `Query` +message for normal operation - that is, the ABCI app developer need not implement +Query functionality if they do not wish to. + +#### Query Proofs + +The CometBFT block header includes a number of hashes, each providing an +anchor for some type of proof about the blockchain. The `ValidatorsHash` enables +quick verification of the validator set, the `DataHash` gives quick +verification of the transactions included in the block. + +The `AppHash` is unique in that it is application specific, and allows for +application-specific Merkle proofs about the state of the application. +While some applications keep all relevant state in the transactions themselves +(like Bitcoin and its UTXOs), others maintain a separated state that is +computed deterministically *from* transactions, but is not contained directly in +the transactions themselves (like Ethereum contracts and accounts). +For such applications, the `AppHash` provides a much more efficient way to verify light-client proofs. + +ABCI applications can take advantage of more efficient light-client proofs for +their state as follows: + +- return the Merkle root of the deterministic application state in + `ResponseFinalizeBlock.Data`. This Merkle root will be included as the `AppHash` in the next block. +- return efficient Merkle proofs about that application state in `ResponseQuery.Proof` + that can be verified using the `AppHash` of the corresponding block. + +For instance, this allows an application's light-client to verify proofs of +absence in the application state, something which is much less efficient to do using the block hash. + +Some applications (eg. Ethereum, Cosmos-SDK) have multiple "levels" of Merkle trees, +where the leaves of one tree are the root hashes of others. To support this, and +the general variability in Merkle proofs, the `ResponseQuery.Proof` has some minimal structure: + +```protobuf +message ProofOps { + repeated ProofOp ops = 1 +} + +message ProofOp { + string type = 1; + bytes key = 2; + bytes data = 3; +} +``` + +Each `ProofOp` contains a proof for a single key in a single Merkle tree, of the specified `type`. +This allows ABCI to support many different kinds of Merkle trees, encoding +formats, and proofs (eg. of presence and absence) just by varying the `type`. +The `data` contains the actual encoded proof, encoded according to the `type`. +When verifying the full proof, the root hash for one ProofOp is the value being +verified for the next ProofOp in the list. The root hash of the final ProofOp in +the list should match the `AppHash` being verified against. + +#### Peer Filtering + +When CometBFT connects to a peer, it sends two queries to the ABCI application +using the following paths, with no additional data: + +- `/p2p/filter/addr/`, where `` denote the IP address and + the port of the connection +- `p2p/filter/id/`, where `` is the peer node ID (ie. the + pubkey.Address() for the peer's PubKey) + +If either of these queries return a non-zero ABCI code, CometBFT will refuse +to connect to the peer. + +#### Paths + +Queries are directed at paths, and may optionally include additional data. + +The expectation is for there to be some number of high level paths +differentiating concerns, like `/p2p`, `/store`, and `/app`. Currently, +CometBFT only uses `/p2p`, for filtering peers. For more advanced use, see the +implementation of +[Query in the Cosmos-SDK](https://github.com/cosmos/cosmos-sdk/blob/e2037f7696fed4fdd4bc076f9e7053fe8178a881/baseapp/abci.go#L557-L565). + +### Crash Recovery + +CometBFT and the application are expected to crash together and there should not +exist a scenario where the application has persisted state of a height greater than the +latest height persisted by CometBFT. + +In practice, persisting the state of a height consists of three steps, the last of which +is the call to the application's `Commit` method, the only place where the application is expected to +persist/commit its state. +On startup (upon recovery), CometBFT calls the `Info` method on the Info Connection to get the latest +committed state of the app. The app MUST return information consistent with the +last block for which it successfully completed `Commit`. + +The three steps performed before the state of a height is considered persisted are: + +- The block is stored by CometBFT in the blockstore +- CometBFT has stored the state returned by the application through `FinalizeBlockResponse` +- The application has committed its state within `Commit`. + +The following diagram depicts the order in which these events happen, and the corresponding +ABCI functions that are called and executed by CometBFT and the application: + + +``` +APP: Execute block Persist application state + / return ResultFinalizeBlock / + / / +Event: ------------- block_stored ------------ / ------------ state_stored --------------- / ----- app_persisted_state + | / | / | +CometBFT: Decide --- Persist block -- Call FinalizeBlock - Persist results ---------- Call Commit -- + on in the (txResults, validator + Block block store updates...) + +``` + +As these three steps are not atomic, we observe different cases based on which steps have been executed +before the crash occurred +(we assume that at least `block_stored` has been executed, otherwise, there is no state persisted, +and the operations for this height are repeated entirely): + +- `block_stored`: we replay `FinalizeBlock` and the steps afterwards. +- `block_stored` and `state_stored`: As the app did not persist its state within `Commit`, we need to re-execute + `FinalizeBlock` to retrieve the results and compare them to the state stored by CometBFT within `state_stored`. + The expected case is that the states will match, otherwise CometBFT panics. +- `block_stored`, `state_stored`, `app_persisted_state`: we move on to the next height. + +Based on the sequence of these events, CometBFT will panic if any of the steps in the sequence happen out of order, +that is if: + +- The application has persisted a block at a height higher than the blocked saved during `state_stored`. +- The `block_stored` step persisted a block at a height smaller than the `state_stored` +- And the difference between the heights of the blocks persisted by `state_stored` and `block_stored` is more +than 1 (this corresponds to a scenario where we stored two blocks in the block store but never persisted the state of the first +block, which should never happen). + +A special case is when a crash happens before the first block is committed - that is, after calling +`InitChain`. In that case, the application's state should still be at height 0 and thus `InitChain` +will be called again. + + +### State Sync + +A new node joining the network can simply join consensus at the genesis height and replay all +historical blocks until it is caught up. However, for large chains this can take a significant +amount of time, often on the order of days or weeks. + +State sync is an alternative mechanism for bootstrapping a new node, where it fetches a snapshot +of the state machine at a given height and restores it. Depending on the application, this can +be several orders of magnitude faster than replaying blocks. + +Note that state sync does not currently backfill historical blocks, so the node will have a +truncated block history - users are advised to consider the broader network implications of this in +terms of block availability and auditability. This functionality may be added in the future. + +For details on the specific ABCI calls and types, see the +[methods](abci%2B%2B_methods.md) section. + +#### Taking Snapshots + +Applications that want to support state syncing must take state snapshots at regular intervals. How +this is accomplished is entirely up to the application. A snapshot consists of some metadata and +a set of binary chunks in an arbitrary format: + +- `Height (uint64)`: The height at which the snapshot is taken. It must be taken after the given + height has been committed, and must not contain data from any later heights. + +- `Format (uint32)`: An arbitrary snapshot format identifier. This can be used to version snapshot + formats, e.g. to switch from Protobuf to MessagePack for serialization. The application can use + this when restoring to choose whether to accept or reject a snapshot. + +- `Chunks (uint32)`: The number of chunks in the snapshot. Each chunk contains arbitrary binary + data, and should be less than 16 MB; 10 MB is a good starting point. + +- `Hash ([]byte)`: An arbitrary hash of the snapshot. This is used to check whether a snapshot is + the same across nodes when downloading chunks. + +- `Metadata ([]byte)`: Arbitrary snapshot metadata, e.g. chunk hashes for verification or any other + necessary info. + +For a snapshot to be considered the same across nodes, all of these fields must be identical. When +sent across the network, snapshot metadata messages are limited to 4 MB. + +When a new node is running state sync and discovering snapshots, CometBFT will query an existing +application via the ABCI `ListSnapshots` method to discover available snapshots, and load binary +snapshot chunks via `LoadSnapshotChunk`. The application is free to choose how to implement this +and which formats to use, but must provide the following guarantees: + +- **Consistent:** A snapshot must be taken at a single isolated height, unaffected by + concurrent writes. This can be accomplished by using a data store that supports ACID + transactions with snapshot isolation. + +- **Asynchronous:** Taking a snapshot can be time-consuming, so it must not halt chain progress, + for example by running in a separate thread. + +- **Deterministic:** A snapshot taken at the same height in the same format must be identical + (at the byte level) across nodes, including all metadata. This ensures good availability of + chunks, and that they fit together across nodes. + +A very basic approach might be to use a datastore with MVCC transactions (such as RocksDB), +start a transaction immediately after block commit, and spawn a new thread which is passed the +transaction handle. This thread can then export all data items, serialize them using e.g. +Protobuf, hash the byte stream, split it into chunks, and store the chunks in the file system +along with some metadata - all while the blockchain is applying new blocks in parallel. + +A more advanced approach might include incremental verification of individual chunks against the +chain app hash, parallel or batched exports, compression, and so on. + +Old snapshots should be removed after some time - generally only the last two snapshots are needed +(to prevent the last one from being removed while a node is restoring it). + +#### Bootstrapping a Node + +An empty node can be state synced by setting the configuration option `statesync.enabled = +true`. The node also needs the chain genesis file for basic chain info, and configuration for +light client verification of the restored snapshot: a set of CometBFT RPC servers, and a +trusted header hash and corresponding height from a trusted source, via the `statesync` +configuration section. + +Once started, the node will connect to the P2P network and begin discovering snapshots. These +will be offered to the local application via the `OfferSnapshot` ABCI method. Once a snapshot +is accepted CometBFT will fetch and apply the snapshot chunks. After all chunks have been +successfully applied, CometBFT verifies the app's `AppHash` against the chain using the light +client, then switches the node to normal consensus operation. + +##### Snapshot Discovery + +When the empty node joins the P2P network, it asks all peers to report snapshots via the +`ListSnapshots` ABCI call (limited to 10 per node). After some time, the node picks the most +suitable snapshot (generally prioritized by height, format, and number of peers), and offers it +to the application via `OfferSnapshot`. The application can choose a number of responses, +including accepting or rejecting it, rejecting the offered format, rejecting the peer who sent +it, and so on. CometBFT will keep discovering and offering snapshots until one is accepted or +the application aborts. + +##### Snapshot Restoration + +Once a snapshot has been accepted via `OfferSnapshot`, CometBFT begins downloading chunks from +any peers that have the same snapshot (i.e. that have identical metadata fields). Chunks are +spooled in a temporary directory, and then given to the application in sequential order via +`ApplySnapshotChunk` until all chunks have been accepted. + +The method for restoring snapshot chunks is entirely up to the application. + +During restoration, the application can respond to `ApplySnapshotChunk` with instructions for how +to continue. This will typically be to accept the chunk and await the next one, but it can also +ask for chunks to be refetched (either the current one or any number of previous ones), P2P peers +to be banned, snapshots to be rejected or retried, and a number of other responses - see the ABCI +reference for details. + +If CometBFT fails to fetch a chunk after some time, it will reject the snapshot and try a +different one via `OfferSnapshot` - the application can choose whether it wants to support +restarting restoration, or simply abort with an error. + +##### Snapshot Verification + +Once all chunks have been accepted, CometBFT issues an `Info` ABCI call to retrieve the +`LastBlockAppHash`. This is compared with the trusted app hash from the chain, retrieved and +verified using the light client. CometBFT also checks that `LastBlockHeight` corresponds to the +height of the snapshot. + +This verification ensures that an application is valid before joining the network. However, the +snapshot restoration may take a long time to complete, so applications may want to employ additional +verification during the restore to detect failures early. This might e.g. include incremental +verification of each chunk against the app hash (using bundled Merkle proofs), checksums to +protect against data corruption by the disk or network, and so on. However, it is important to +note that the only trusted information available is the app hash, and all other snapshot metadata +can be spoofed by adversaries. + +Apps may also want to consider state sync denial-of-service vectors, where adversaries provide +invalid or harmful snapshots to prevent nodes from joining the network. The application can +counteract this by asking CometBFT to ban peers. As a last resort, node operators can use +P2P configuration options to whitelist a set of trusted peers that can provide valid snapshots. + +##### Transition to Consensus + +Once the snapshots have all been restored, CometBFT gathers additional information necessary for +bootstrapping the node (e.g. chain ID, consensus parameters, validator sets, and block headers) +from the genesis file and light client RPC servers. It also calls `Info` to verify the following: + +- that the app hash from the snapshot it has delivered to the Application matches the apphash + stored in the next height's block + +- that the version that the Application returns in `ResponseInfo` matches the version in the + current height's block header + +Once the state machine has been restored and CometBFT has gathered this additional +information, it transitions to consensus. As of ABCI 2.0, CometBFT ensures the necessary conditions +to switch are met [RFC-100](https://github.com/cometbft/cometbft/blob/v0.38.x/docs/rfc/rfc-100-abci-vote-extension-propag.md#base-implementation-persist-and-propagate-extended-commit-history). +From the application's point of view, these operations are transparent, unless the application has just upgraded to ABCI 2.0. +In that case, the application needs to be properly configured and aware of certain constraints in terms of when +to provide vote extensions. More details can be found in the section below. + +Once a node switches to consensus, it operates like any other node, apart from having a truncated block history at the height of the restored snapshot. + +## Application configuration required to switch to ABCI 2.0 + +Introducing vote extensions requires changes to the configuration of the application. + +First of all, switching to a version of CometBFT with vote extensions, requires a coordinated upgrade. +For a detailed description on the upgrade path, please refer to the corresponding +[section](https://github.com/cometbft/cometbft/blob/v0.38.x/docs/rfc/rfc-100-abci-vote-extension-propag.md#upgrade-path) in RFC-100. + +There is a newly introduced [**consensus parameter**](./abci%2B%2B_app_requirements.md#abciparamsvoteextensionsenableheight): `VoteExtensionsEnableHeight`. +This parameter represents the height at which vote extensions are +required for consensus to proceed, with 0 being the default value (no vote extensions). +A chain can enable vote extensions either: + +- at genesis by setting `VoteExtensionsEnableHeight` to be equal, e.g., to the `InitialHeight` +- or via the application logic by changing the `ConsensusParam` to configure the +`VoteExtensionsEnableHeight`. + +Once the (coordinated) upgrade to ABCI 2.0 has taken place, at height *hu*, +the value of `VoteExtensionsEnableHeight` MAY be set to some height, *he*, +which MUST be higher than the current height of the chain. Thus the earliest value for + *he* is *hu* + 1. + +Once a node reaches the configured height, +for all heights *h ≥ he*, the consensus algorithm will +reject as invalid any precommit messages that do not have signed vote extension data. +If the application requires it, a 0-length vote extension is allowed, but it MUST be signed +and present in the precommit message. +Likewise, for all heights *h < he*, any precommit messages that *do* have vote extensions +will also be rejected as malformed. +Height *he* is somewhat special, as calls to `PrepareProposal` MUST NOT +have vote extension data, but all precommit votes in that height MUST carry a vote extension, +even if the extension is `nil`. +Height *he + 1* is the first height for which `PrepareProposal` MUST have vote +extension data and all precommit votes in that height MUST have a vote extension. + +Corollary, [CometBFT will decide](./abci%2B%2B_comet_expected_behavior.md#handling-upgrades-to-abci-20) which data to store, and require for successful operations, based on the current height +of the chain. diff --git a/cometbft/v0.38/spec/blockchain/Blockchain.mdx b/cometbft/v0.38/spec/blockchain/Blockchain.mdx new file mode 100644 index 00000000..ae9119df --- /dev/null +++ b/cometbft/v0.38/spec/blockchain/Blockchain.mdx @@ -0,0 +1,14 @@ +--- +order: 1 +parent: + title: Blockchain + order: false +--- + +# Blockchain + +This section describes the core types and functionality of the CometBFT protocol implementation. + +[Core Data Structures](/cometbft/v0.38/spec/core/Data_structures) +[Encoding](/cometbft/v0.38/spec/core/encoding) +[State](/cometbft/v0.38/spec/core/state) diff --git a/cometbft/v0.38/spec/blockchain/blockchain.md b/cometbft/v0.38/spec/blockchain/blockchain.md new file mode 100644 index 00000000..fcc080ee --- /dev/null +++ b/cometbft/v0.38/spec/blockchain/blockchain.md @@ -0,0 +1,3 @@ +# Blockchain + +Deprecated see [core/data_structures.md](../core/data_structures.md) diff --git a/cometbft/v0.38/spec/blockchain/encoding.md b/cometbft/v0.38/spec/blockchain/encoding.md new file mode 100644 index 00000000..aa2c9ab3 --- /dev/null +++ b/cometbft/v0.38/spec/blockchain/encoding.md @@ -0,0 +1,3 @@ +# Encoding + +Deprecated see [core/data_structures.md](../core/encoding.md) diff --git a/cometbft/v0.38/spec/blockchain/state.md b/cometbft/v0.38/spec/blockchain/state.md new file mode 100644 index 00000000..f4f1d952 --- /dev/null +++ b/cometbft/v0.38/spec/blockchain/state.md @@ -0,0 +1,3 @@ +# State + +Deprecated see [core/state.md](../core/state.md) diff --git a/cometbft/v0.38/spec/consensus/BFT-Time.mdx b/cometbft/v0.38/spec/consensus/BFT-Time.mdx new file mode 100644 index 00000000..9312d9b3 --- /dev/null +++ b/cometbft/v0.38/spec/consensus/BFT-Time.mdx @@ -0,0 +1,56 @@ +--- +order: 2 +--- +# BFT Time + +CometBFT provides a deterministic, Byzantine fault-tolerant, source of time. +Time in CometBFT is defined with the Time field of the block header. + +It satisfies the following properties: + +- Time Monotonicity: Time is monotonically increasing, i.e., given +a header H1 for height h1 and a header H2 for height `h2 = h1 + 1`, `H1.Time < H2.Time`. +- Time Validity: Given a set of Commit votes that forms the `block.LastCommit` field, a range of +valid values for the Time field of the block header is defined only by +Precommit messages (from the LastCommit field) sent by correct processes, i.e., +a faulty process cannot arbitrarily increase the Time value. + +In the context of CometBFT, time is of type int64 and denotes UNIX time in milliseconds, i.e., +corresponds to the number of milliseconds since January 1, 1970. +Before defining rules that need to be enforced by Tendermint, the consensus algorithm adopted in CometBFT, +so the properties above holds, we introduce the following definition: + +- median of a Commit is equal to the median of `Vote.Time` fields of the `Vote` messages, +where the value of `Vote.Time` is counted number of times proportional to the process voting power. As +the voting power is not uniform (one process one vote), a vote message is actually an aggregator of the same votes whose +number is equal to the voting power of the process that has casted the corresponding votes message. + +Let's consider the following example: + +- we have four processes p1, p2, p3 and p4, with the following voting power distribution (p1, 23), (p2, 27), (p3, 10) +and (p4, 10). The total voting power is 70 (`N = 3f+1`, where `N` is the total voting power, and `f` is the maximum voting +power of the faulty processes), so we assume that the faulty processes have at most 23 of voting power. +Furthermore, we have the following vote messages in some LastCommit field (we ignore all fields except Time field): + - (p1, 100), (p2, 98), (p3, 1000), (p4, 500). We assume that p3 and p4 are faulty processes. Let's assume that the + `block.LastCommit` message contains votes of processes p2, p3 and p4. Median is then chosen the following way: + the value 98 is counted 27 times, the value 1000 is counted 10 times and the value 500 is counted also 10 times. + So the median value will be the value 98. No matter what set of messages with at least `2f+1` voting power we + choose, the median value will always be between the values sent by correct processes. + +We ensure Time Monotonicity and Time Validity properties by the following rules: + +- let rs denotes `RoundState` (consensus internal state) of some process. Then +`rs.ProposalBlock.Header.Time == median(rs.LastCommit) && +rs.Proposal.Timestamp == rs.ProposalBlock.Header.Time`. + +- Furthermore, when creating the `vote` message, the following rules for determining `vote.Time` field should hold: + + - if `rs.LockedBlock` is defined then + `vote.Time = max(rs.LockedBlock.Timestamp + time.Millisecond, time.Now())`, where `time.Now()` + denotes local Unix time in milliseconds + + - else if `rs.Proposal` is defined then + `vote.Time = max(rs.Proposal.Timestamp + time.Millisecond,, time.Now())`, + + - otherwise, `vote.Time = time.Now())`. In this case vote is for `nil` so it is not taken into account for + the timestamp of the next block. diff --git a/cometbft/v0.38/spec/consensus/Byzantine-Consensus-Algorithm.mdx b/cometbft/v0.38/spec/consensus/Byzantine-Consensus-Algorithm.mdx new file mode 100644 index 00000000..9fe30bf9 --- /dev/null +++ b/cometbft/v0.38/spec/consensus/Byzantine-Consensus-Algorithm.mdx @@ -0,0 +1,350 @@ +--- +order: 1 +--- +# Byzantine Consensus Algorithm + +## Terms + +- The network is composed of optionally connected _nodes_. Nodes + directly connected to a particular node are called _peers_. +- The consensus process in deciding the next block (at some _height_ + `H`) is composed of one or many _rounds_. +- `NewHeight`, `Propose`, `Prevote`, `Precommit`, and `Commit` + represent state machine states of a round. (aka `RoundStep` or + just "step"). +- A node is said to be _at_ a given height, round, and step, or at + `(H,R,S)`, or at `(H,R)` in short to omit the step. +- To _prevote_ or _precommit_ something means to broadcast a prevote + or precommit [vote](https://github.com/cometbft/cometbft/blob/af3bc47df982e271d4d340a3c5e0d773e440466d/types/vote.go#L50) + for something. +- A vote _at_ `(H,R)` is a vote signed with the bytes for `H` and `R` + included in its [sign-bytes](../core/data_structures.md#vote). +- _+2/3_ is short for "more than 2/3" +- _1/3+_ is short for "1/3 or more" +- A set of +2/3 of prevotes for a particular block or `` at + `(H,R)` is called a _proof-of-lock-change_ or _PoLC_ for short. + +## State Machine Overview + +At each height of the blockchain a round-based protocol is run to +determine the next block. Each round is composed of three _steps_ +(`Propose`, `Prevote`, and `Precommit`), along with two special steps +`Commit` and `NewHeight`. + +In the optimal scenario, the order of steps is: + +```md +NewHeight -> (Propose -> Prevote -> Precommit)+ -> Commit -> NewHeight ->... +``` + +The sequence `(Propose -> Prevote -> Precommit)` is called a _round_. +There may be more than one round required to commit a block at a given +height. Examples for why more rounds may be required include: + +- The designated proposer was not online. +- The block proposed by the designated proposer was not valid. +- The block proposed by the designated proposer did not propagate + in time. +- The block proposed was valid, but +2/3 of prevotes for the proposed + block were not received in time for enough validator nodes by the + time they reached the `Precommit` step. Even though +2/3 of prevotes + are necessary to progress to the next step, at least one validator + may have voted `` or maliciously voted for something else. +- The block proposed was valid, and +2/3 of prevotes were received for + enough nodes, but +2/3 of precommits for the proposed block were not + received for enough validator nodes. + +Some of these problems are resolved by moving onto the next round & +proposer. Others are resolved by increasing certain round timeout +parameters over each successive round. + +## State Machine Diagram + +```md + +-------------------------------------+ + v |(Wait til `CommmitTime+timeoutCommit`) + +-----------+ +-----+-----+ + +----------> | Propose +--------------+ | NewHeight | + | +-----------+ | +-----------+ + | | ^ + |(Else, after timeoutPrecommit) v | ++-----+-----+ +-----------+ | +| Precommit | <------------------------+ Prevote | | ++-----+-----+ +-----------+ | + |(When +2/3 Precommits for block found) | + v | ++--------------------------------------------------------------------+ +| Commit | +| | +| * Set CommitTime = now; | +| * Wait for block, then stage/save/commit block; | ++--------------------------------------------------------------------+ +``` + +# Background Gossip + +A node may not have a corresponding validator private key, but it +nevertheless plays an active role in the consensus process by relaying +relevant meta-data, proposals, blocks, and votes to its peers. A node +that has the private keys of an active validator and is engaged in +signing votes is called a _validator-node_. All nodes (not just +validator-nodes) have an associated state (the current height, round, +and step) and work to make progress. + +Between two nodes there exists a `Connection`, and multiplexed on top of +this connection are fairly throttled `Channel`s of information. An +epidemic gossip protocol is implemented among some of these channels to +bring peers up to speed on the most recent state of consensus. For +example, + +- Nodes gossip `PartSet` parts of the current round's proposer's + proposed block. A LibSwift inspired algorithm is used to quickly + broadcast blocks across the gossip network. +- Nodes gossip prevote/precommit votes. A node `NODE_A` that is ahead + of `NODE_B` can send `NODE_B` prevotes or precommits for `NODE_B`'s + current (or future) round to enable it to progress forward. +- Nodes gossip prevotes for the proposed PoLC (proof-of-lock-change) + round if one is proposed. +- Nodes gossip to nodes lagging in blockchain height with block + [commits](https://github.com/cometbft/cometbft/blob/af3bc47df982e271d4d340a3c5e0d773e440466d/types/block.go#L738) + for older blocks. +- Nodes opportunistically gossip `ReceivedVote` messages to hint peers what + votes it already has. +- Nodes broadcast their current state to all neighboring peers. (but + is not gossiped further) + +There's more, but let's not get ahead of ourselves here. + +## Proposals + +A proposal is signed and published by the designated proposer at each +round. The proposer is chosen by a deterministic and non-choking round +robin selection algorithm that selects proposers in proportion to their +voting power (see +[implementation](https://github.com/cometbft/cometbft/blob/af3bc47df982e271d4d340a3c5e0d773e440466d/types/validator_set.go#L51)). + +A proposal at `(H,R)` is composed of a block and an optional latest +`PoLC-Round < R` which is included iff the proposer knows of one. This +hints the network to allow nodes to unlock (when safe) to ensure the +liveness property. + +## State Machine Spec + +### Propose Step (height:H,round:R) + +Upon entering `Propose`: + +- The designated proposer proposes a block at `(H,R)`. + +The `Propose` step ends: + +- After `timeoutProposeR` after entering `Propose`. --> goto + `Prevote(H,R)` +- After receiving proposal block and all prevotes at `PoLC-Round`. --> + goto `Prevote(H,R)` +- After [common exit conditions](#common-exit-conditions) + +### Prevote Step (height:H,round:R) + +Upon entering `Prevote`, each validator broadcasts its prevote vote. + +- First, if the validator is locked on a block since `LastLockRound` + but now has a PoLC for something else at round `PoLC-Round` where + `LastLockRound < PoLC-Round < R`, then it unlocks. +- If the validator is still locked on a block, it prevotes that. +- Else, if the proposed block from `Propose(H,R)` is good, it + prevotes that. +- Else, if the proposal is invalid or wasn't received on time, it + prevotes ``. + +The `Prevote` step ends: + +- After +2/3 prevotes for a particular block or ``. -->; goto + `Precommit(H,R)` +- After `timeoutPrevote` after receiving any +2/3 prevotes. --> goto + `Precommit(H,R)` +- After [common exit conditions](#common-exit-conditions) + +### Precommit Step (height:H,round:R) + +Upon entering `Precommit`, each validator broadcasts its precommit vote. + +- If the validator has a PoLC at `(H,R)` for a particular block `B`, it + (re)locks (or changes lock to) and precommits `B` and sets + `LastLockRound = R`. +- Else, if the validator has a PoLC at `(H,R)` for ``, it unlocks + and precommits ``. +- Else, it keeps the lock unchanged and precommits ``. + +A precommit for `` means "I didn’t see a PoLC for this round, but I +did get +2/3 prevotes and waited a bit". + +The Precommit step ends: + +- After +2/3 precommits for ``. --> goto `Propose(H,R+1)` +- After `timeoutPrecommit` after receiving any +2/3 precommits. --> goto + `Propose(H,R+1)` +- After [common exit conditions](#common-exit-conditions) + +### Common exit conditions + +- After +2/3 precommits for a particular block. --> goto + `Commit(H)` +- After any +2/3 prevotes received at `(H,R+x)`. --> goto + `Prevote(H,R+x)` +- After any +2/3 precommits received at `(H,R+x)`. --> goto + `Precommit(H,R+x)` + +### Commit Step (height:H) + +- Set `CommitTime = now()` +- Wait until block is received. --> goto `NewHeight(H+1)` + +### NewHeight Step (height:H) + +- Move `Precommits` to `LastCommit` and increment height. +- Set `StartTime = CommitTime+timeoutCommit` +- Wait until `StartTime` to receive straggler commits. --> goto + `Propose(H,0)` + +## Proofs + +### Proof of Safety + +Assume that at most -1/3 of the voting power of validators is byzantine. +If a validator commits block `B` at round `R`, it's because it saw +2/3 +of precommits at round `R`. This implies that 1/3+ of honest nodes are +still locked at round `R' > R`. These locked validators will remain +locked until they see a PoLC at `R' > R`, but this won't happen because +1/3+ are locked and honest, so at most -2/3 are available to vote for +anything other than `B`. + +### Proof of Liveness + +If 1/3+ honest validators are locked on two different blocks from +different rounds, a proposers' `PoLC-Round` will eventually cause nodes +locked from the earlier round to unlock. Eventually, the designated +proposer will be one that is aware of a PoLC at the later round. Also, +`timeoutProposalR` increments with round `R`, while the size of a +proposal are capped, so eventually the network is able to "fully gossip" +the whole proposal (e.g. the block & PoLC). + +### Proof of Fork Accountability + +Define the JSet (justification-vote-set) at height `H` of a validator +`V1` to be all the votes signed by the validator at `H` along with +justification PoLC prevotes for each lock change. For example, if `V1` +signed the following precommits: `Precommit(B1 @ round 0)`, +`Precommit( @ round 1)`, `Precommit(B2 @ round 4)` (note that no +precommits were signed for rounds 2 and 3, and that's ok), +`Precommit(B1 @ round 0)` must be justified by a PoLC at round 0, and +`Precommit(B2 @ round 4)` must be justified by a PoLC at round 4; but +the precommit for `` at round 1 is not a lock-change by definition +so the JSet for `V1` need not include any prevotes at round 1, 2, or 3 +(unless `V1` happened to have prevoted for those rounds). + +Further, define the JSet at height `H` of a set of validators `VSet` to +be the union of the JSets for each validator in `VSet`. For a given +commit by honest validators at round `R` for block `B` we can construct +a JSet to justify the commit for `B` at `R`. We say that a JSet +_justifies_ a commit at `(H,R)` if all the committers (validators in the +commit-set) are each justified in the JSet with no duplicitous vote +signatures (by the committers). + +- **Lemma**: When a fork is detected by the existence of two + conflicting [commits](../core/data_structures.md#commit), the + union of the JSets for both commits (if they can be compiled) must + include double-signing by at least 1/3+ of the validator set. + **Proof**: The commit cannot be at the same round, because that + would immediately imply double-signing by 1/3+. Take the union of + the JSets of both commits. If there is no double-signing by at least + 1/3+ of the validator set in the union, then no honest validator + could have precommitted any different block after the first commit. + Yet, +2/3 did. Reductio ad absurdum. + +As a corollary, when there is a fork, an external process can determine +the blame by requiring each validator to justify all of its round votes. +Either we will find 1/3+ who cannot justify at least one of their votes, +and/or, we will find 1/3+ who had double-signed. + +### Alternative algorithm + +Alternatively, we can take the JSet of a commit to be the "full commit". +That is, if light clients and validators do not consider a block to be +committed unless the JSet of the commit is also known, then we get the +desirable property that if there ever is a fork (e.g. there are two +conflicting "full commits"), then 1/3+ of the validators are immediately +punishable for double-signing. + +There are many ways to ensure that the gossip network efficiently share +the JSet of a commit. One solution is to add a new message type that +tells peers that this node has (or does not have) a +2/3 majority for B +(or) at (H,R), and a bitarray of which votes contributed towards that +majority. Peers can react by responding with appropriate votes. + +We will implement such an algorithm for the next iteration of the +consensus protocol. + +Other potential improvements include adding more data in votes such as +the last known PoLC round that caused a lock change, and the last voted +round/step (or, we may require that validators not skip any votes). This +may make JSet verification/gossip logic easier to implement. + +### Censorship Attacks + +Due to the definition of a block +[commit](https://github.com/cometbft/cometbft/blob/v0.38.x/docs/core/validators.md), any 1/3+ coalition of +validators can halt the blockchain by not broadcasting their votes. Such +a coalition can also censor particular transactions by rejecting blocks +that include these transactions, though this would result in a +significant proportion of block proposals to be rejected, which would +slow down the rate of block commits of the blockchain, reducing its +utility and value. The malicious coalition might also broadcast votes in +a trickle so as to grind blockchain block commits to a near halt, or +engage in any combination of these attacks. + +If a global active adversary were also involved, it can partition the +network in such a way that it may appear that the wrong subset of +validators were responsible for the slowdown. This is not just a +limitation of Tendermint, but rather a limitation of all consensus +protocols whose network is potentially controlled by an active +adversary. + +### Overcoming Forks and Censorship Attacks + +For these types of attacks, a subset of the validators through external +means should coordinate to sign a reorg-proposal that chooses a fork +(and any evidence thereof) and the initial subset of validators with +their signatures. Validators who sign such a reorg-proposal forego its +collateral on all other forks. Clients should verify the signatures on +the reorg-proposal, verify any evidence, and make a judgement or prompt +the end-user for a decision. For example, a phone wallet app may prompt +the user with a security warning, while a refrigerator may accept any +reorg-proposal signed by +1/2 of the original validators. + +No non-synchronous Byzantine fault-tolerant algorithm can come to +consensus when 1/3+ of validators are dishonest, yet a fork assumes that +1/3+ of validators have already been dishonest by double-signing or +lock-changing without justification. So, signing the reorg-proposal is a +coordination problem that cannot be solved by any non-synchronous +protocol (i.e. automatically, and without making assumptions about the +reliability of the underlying network). It must be provided by means +external to the weakly-synchronous Tendermint consensus algorithm. For +now, we leave the problem of reorg-proposal coordination to human +coordination via internet media. Validators must take care to ensure +that there are no significant network partitions, to avoid situations +where two conflicting reorg-proposals are signed. + +Assuming that the external coordination medium and protocol is robust, +it follows that forks are less of a concern than [censorship +attacks](#censorship-attacks). + +### Canonical vs subjective commit + +We distinguish between "canonical" and "subjective" commits. A subjective commit is what +each validator sees locally when they decide to commit a block. The canonical commit is +what is included by the proposer of the next block in the `LastCommit` field of +the block. This is what makes it canonical and ensures every validator agrees on the canonical commit, +even if it is different from the +2/3 votes a validator has seen, which caused the validator to +commit the respective block. Each block contains a canonical +2/3 commit for the previous +block. diff --git a/cometbft/v0.38/spec/consensus/Consensus-Paper.mdx b/cometbft/v0.38/spec/consensus/Consensus-Paper.mdx new file mode 100644 index 00000000..d3d71b76 --- /dev/null +++ b/cometbft/v0.38/spec/consensus/Consensus-Paper.mdx @@ -0,0 +1,28 @@ +--- +order: 1 +--- + +# Consensus Paper + +The repository contains the specification (and the proofs) of the Tendermint +consensus protocol, adopted in CometBFT. + +## How to install Latex on MacOS + +MacTex is Latex distribution for MacOS. You can download it [here](http://www.tug.org/mactex/mactex-download.html). + +Popular IDE for Latex-based projects is TexStudio. It can be downloaded +[here](https://www.texstudio.org/). + +## How to build project + +In order to compile the latex files (and write bibliography), execute + +`$ pdflatex paper`
+`$ bibtex paper`
+`$ pdflatex paper`
+`$ pdflatex paper`
+ +The generated file is paper.pdf. You can open it with + +`$ open paper.pdf` diff --git a/cometbft/v0.38/spec/consensus/Creating-Proposal.mdx b/cometbft/v0.38/spec/consensus/Creating-Proposal.mdx new file mode 100644 index 00000000..feeb8e59 --- /dev/null +++ b/cometbft/v0.38/spec/consensus/Creating-Proposal.mdx @@ -0,0 +1,61 @@ +--- +order: 2 +--- +# Creating a proposal + +A block consists of a header, transactions, votes (the commit), +and a list of evidence of malfeasance (eg. signing conflicting votes). + +Outstanding evidence items get priority over outstanding transactions in the mempool. +All in all, the block MUST NOT exceed `ConsensusParams.Block.MaxBytes`, +or 100MB if `ConsensusParams.Block.MaxBytes == -1`. + +## Reaping transactions from the mempool + +When we reap transactions from the mempool, we calculate maximum data +size by subtracting maximum header size (`MaxHeaderBytes`), the maximum +protobuf overhead for a block (`MaxOverheadForBlock`), the size of +the last commit (if present) and evidence (if present). While reaping +we account for protobuf overhead for each transaction. + +```go +func MaxDataBytes(maxBytes, evidenceBytes int64, valsCount int) int64 { + return maxBytes - + MaxOverheadForBlock - + MaxHeaderBytes - + MaxCommitBytes(valsCount) - + evidenceBytes +} +``` + +If `ConsensusParams.Block.MaxBytes == -1`, we reap *all* outstanding transactions from the mempool + +## Preparing the proposal + +Once the transactions have been reaped from the mempool according to the rules described above, +CometBFT calls `PrepareProposal` to the application with the transaction list that has just been reaped. +As part of this call the application can remove, add, or reorder transactions in the transaction list. + +The `RequestPrepareProposal` contains two important fields: + +* `MaxTxBytes`, which contains the value returned by `MaxDataBytes` described above. + The application MUST NOT return a list of transactions whose size exceeds this number. +* `Txs`, which contains the list of reaped transactions. + +For more details on `PrepareProposal`, please see the +[relevant part of the spec](../abci/abci%2B%2B_methods.md#prepareproposal) + +## Validating transactions in the mempool + +Before we accept a transaction in the mempool, we check if its size is no more +than {MaxDataSize}. {MaxDataSize} is calculated using the same formula as +above, except we assume there is no evidence. + +```go +func MaxDataBytesNoEvidence(maxBytes int64, valsCount int) int64 { + return maxBytes - + MaxOverheadForBlock - + MaxHeaderBytes - + MaxCommitBytes(valsCount) +} +``` diff --git a/cometbft/v0.38/spec/consensus/Evidence.mdx b/cometbft/v0.38/spec/consensus/Evidence.mdx new file mode 100644 index 00000000..ad341f28 --- /dev/null +++ b/cometbft/v0.38/spec/consensus/Evidence.mdx @@ -0,0 +1,203 @@ +--- +order: 4 +--- + +# Evidence + +Evidence is an important component of CometBFT's security model. Whilst the core +consensus protocol provides correctness gaurantees for state machine replication +that can tolerate less than 1/3 failures, the evidence system looks to detect and +gossip byzantine faults whose combined power is greater than or equal to 1/3. It is worth noting that +the evidence system is designed purely to detect possible attacks, gossip them, +commit them on chain and inform the application running on top of CometBFT. +Evidence in itself does not punish "bad actors", this is left to the discretion +of the application. A common form of punishment is slashing where the validators +that were caught violating the protocol have all or a portion of their voting +power removed. Evidence, given the assumption that 1/3+ of the network is still +byzantine, is susceptible to censorship and should therefore be considered added +security on a "best effort" basis. + +This document walks through the various forms of evidence, how they are detected, +gossiped, verified and committed. + +> NOTE: Evidence here is internal to CometBFT and should not be confused with +> application evidence + +## Detection + +### Equivocation + +Equivocation is the most fundamental of byzantine faults. Simply put, to prevent +replication of state across all nodes, a validator tries to convince some subset +of nodes to commit one block whilst convincing another subset to commit a +different block. This is achieved by double voting (hence +`DuplicateVoteEvidence`). A successful duplicate vote attack requires greater +than 1/3 voting power and a (temporary) network partition between the aforementioned +subsets. This is because in consensus, votes are gossiped around. When a node +observes two conflicting votes from the same peer, it will use the two votes of +evidence and begin gossiping this evidence to other nodes. [Verification](#duplicatevoteevidence) is addressed further down. + +```go +type DuplicateVoteEvidence struct { + VoteA Vote + VoteB Vote + + // and abci specific fields +} +``` + +### Light Client Attacks + +Light clients also comply with the 1/3+ security model, however, by using a +different, more lightweight verification method they are subject to a +different kind of 1/3+ attack whereby the byzantine validators could sign an +alternative light block that the light client will think is valid. Detection, +explained in greater detail +[here](../light-client/detection/detection_003_reviewed.md), involves comparison +with multiple other nodes in the hope that at least one is "honest". An "honest" +node will return a challenging light block for the light client to validate. If +this challenging light block also meets the +[validation criteria](../light-client/verification/verification_001_published.md) +then the light client sends the "forged" light block to the node. +[Verification](#lightclientattackevidence) is addressed further down. + +```go +type LightClientAttackEvidence struct { + ConflictingBlock LightBlock + CommonHeight int64 + + // and abci specific fields +} +``` + +## Verification + +If a node receives evidence, it will first try to verify it, then persist it. +Evidence of byzantine behavior should only be committed once (uniqueness) and +should be committed within a certain period from the point that it occurred +(timely). Timelines is defined by the `EvidenceParams`: `MaxAgeNumBlocks` and +`MaxAgeDuration`. In Proof of Stake chains where validators are bonded, evidence +age should be less than the unbonding period so validators still can be +punished. Given these two propoerties the following initial checks are made. + +1. Has the evidence expired? This is done by taking the height of the `Vote` + within `DuplicateVoteEvidence` or `CommonHeight` within + `LightClientAttakEvidence`. The evidence height is then used to retrieve the + header and thus the time of the block that corresponds to the evidence. If + `CurrentHeight - MaxAgeNumBlocks > EvidenceHeight` && `CurrentTime - + MaxAgeDuration > EvidenceTime`, the evidence is considered expired and + ignored. + +2. Has the evidence already been committed? The evidence pool tracks the hash of + all committed evidence and uses this to determine uniqueness. If a new + evidence has the same hash as a committed one, the new evidence will be + ignored. + +### DuplicateVoteEvidence + +Valid `DuplicateVoteEvidence` must adhere to the following rules: + +- Validator Address, Height, Round and Type must be the same for both votes + +- BlockID must be different for both votes (BlockID can be for a nil block) + +- Validator must have been in the validator set at that height + +- Vote signature must be correctly signed. This also uses `ChainID` so we know + that the fault occurred on this chain + +### LightClientAttackEvidence + +Valid Light Client Attack Evidence must adhere to the following rules: + +- If the header of the light block is invalid, thus indicating a lunatic attack, + the node must check that they can use `verifySkipping` from their header at + the common height to the conflicting header + +- If the header is valid, then the validator sets are the same and this is + either a form of equivocation or amnesia. We therefore check that 2/3 of the + validator set also signed the conflicting header. + +- The nodes own header at the same height as the conflicting header must have a + different hash to the conflicting header. + +- If the nodes latest header is less in height to the conflicting header, then + the node must check that the conflicting block has a time that is less than + this latest header (This is a forward lunatic attack). + +## Gossiping + +If a node verifies evidence it then broadcasts it to all peers, continously sending +the same evidence once every 10 seconds until the evidence is seen on chain or +expires. + +## Commiting on Chain + +Evidence takes strict priority over regular transactions, thus a block is filled +with evidence first and transactions take up the remainder of the space. To +mitigate the threat of an already punished node from spamming the network with +more evidence, the size of the evidence in a block can be capped by +`EvidenceParams.MaxBytes`. Nodes receiving blocks with evidence will validate +the evidence before sending `Prevote` and `Precommit` votes. The evidence pool +will usually cache verifications so that this process is much quicker. + +## Sending Evidence to the Application + +After evidence is committed, the block is then processed by the block executor +which delivers the evidence to the application via `EndBlock`. Evidence is +stripped of the actual proof, split up per faulty validator and only the +validator, height, time and evidence type is sent. + +```proto +enum EvidenceType { + UNKNOWN = 0; + DUPLICATE_VOTE = 1; + LIGHT_CLIENT_ATTACK = 2; +} + +message Evidence { + EvidenceType type = 1; + // The offending validator + Validator validator = 2 [(gogoproto.nullable) = false]; + // The height when the offense occurred + int64 height = 3; + // The corresponding time where the offense occurred + google.protobuf.Timestamp time = 4 [ + (gogoproto.nullable) = false, (gogoproto.stdtime) = true]; + // Total voting power of the validator set in case the ABCI application does + // not store historical validators. + // https://github.com/tendermint/tendermint/issues/4581 + int64 total_voting_power = 5; +} +``` + +`DuplicateVoteEvidence` and `LightClientAttackEvidence` are self-contained in +the sense that the evidence can be used to derive the `abci.Evidence` that is +sent to the application. Because of this, extra fields are necessary: + +```go +type DuplicateVoteEvidence struct { + VoteA *Vote + VoteB *Vote + + // abci specific information + TotalVotingPower int64 + ValidatorPower int64 + Timestamp time.Time +} + +type LightClientAttackEvidence struct { + ConflictingBlock *LightBlock + CommonHeight int64 + + // abci specific information + ByzantineValidators []*Validator + TotalVotingPower int64 + Timestamp time.Time +} +``` + +These ABCI specific fields don't affect validity of the evidence itself but must +be consistent amongst nodes and agreed upon on chain. If evidence with the +incorrect abci information is sent, a node will create new evidence from it and +replace the ABCI fields with the correct information. diff --git a/cometbft/v0.38/spec/consensus/Light-Client.mdx b/cometbft/v0.38/spec/consensus/Light-Client.mdx new file mode 100644 index 00000000..160e3efc --- /dev/null +++ b/cometbft/v0.38/spec/consensus/Light-Client.mdx @@ -0,0 +1,10 @@ +--- +order: 1 +parent: + title: Light Client + order: false +--- + +# Light Client Protocol + +Deprecated, please see [light-client](/cometbft/v0.38/spec/light-client/Light-Client-Specification). diff --git a/cometbft/v0.38/spec/consensus/Overview.mdx b/cometbft/v0.38/spec/consensus/Overview.mdx new file mode 100644 index 00000000..6036b7b1 --- /dev/null +++ b/cometbft/v0.38/spec/consensus/Overview.mdx @@ -0,0 +1,29 @@ +--- +order: 1 +parent: + title: Consensus + order: 4 +--- + +# Consensus + +Specification of the consensus protocol implemented in CometBFT. + +## Contents + +- [Consensus Paper](/cometbft/v0.38/spec/consensus/Consensus-Paper) - Latex paper on + [arxiv](https://arxiv.org/abs/1807.04938) describing the + Tendermint consensus algorithm, adopted in CometBFT, with proofs of safety and termination. +- [BFT Time](/cometbft/v0.38/spec/consensus/BFT-Time) - How the timestamp in a CometBFT + block header is computed in a Byzantine Fault Tolerant manner +- [Creating Proposal](/cometbft/v0.38/spec/consensus/Creating-Proposal) - How a proposer + creates a block proposal for consensus +- [Light Client Protocol](/cometbft/v0.38/spec/consensus/Light-Client) - A protocol for light weight consensus + verification and syncing to the latest state +- [Validator Signing](/cometbft/v0.38/spec/consensus/Validator-Signing) - Rules for cryptographic signatures + produced by validators. +- [Write Ahead Log](/cometbft/v0.38/spec/consensus/WAL) - Write ahead log used by the + consensus state machine to recover from crashes. + +There is also a [stale markdown description](/cometbft/v0.38/spec/consensus/Byzantine-Consensus-Algorithm) of the consensus state machine +(TODO update this). diff --git a/cometbft/v0.38/spec/consensus/Proposer-Selection.mdx b/cometbft/v0.38/spec/consensus/Proposer-Selection.mdx new file mode 100644 index 00000000..e5142bd3 --- /dev/null +++ b/cometbft/v0.38/spec/consensus/Proposer-Selection.mdx @@ -0,0 +1,323 @@ +--- +order: 3 +--- + +# Proposer Selection Procedure + +This document specifies the Proposer Selection Procedure that is used in Tendermint, the consensus algorithm adopted in CometBFT, to choose a round proposer. +As Tendermint is “leader-based consensus protocol”, the proposer selection is critical for its correct functioning. + +At a given block height, the proposer selection algorithm runs with the same validator set at each round . +Between heights, an updated validator set may be specified by the application as part of the ABCIResponses' EndBlock. + +## Requirements for Proposer Selection + +This sections covers the requirements with Rx being mandatory and Ox optional requirements. +The following requirements must be met by the Proposer Selection procedure: + +### R1: Determinism + +Given a validator set `V`, and two honest validators `p` and `q`, for each height `h` and each round `r` the following must hold: + + `proposer_p(h,r) = proposer_q(h,r)` + +where `proposer_p(h,r)` is the proposer returned by the Proposer Selection Procedure at process `p`, at height `h` and round `r`. + +### R2: Fairness + +Given a validator set with total voting power P and a sequence S of elections. In any sub-sequence of S with length C*P, a validator v must be elected as proposer P/VP(v) times, i.e. with frequency: + + f(v) ~ VP(v) / P + +where C is a tolerance factor for validator set changes with following values: + +- C == 1 if there are no validator set changes +- C ~ k when there are validator changes + +*[this needs more work]* + +## Basic Algorithm + +At its core, the proposer selection procedure uses a weighted round-robin algorithm. + +A model that gives a good intuition on how/ why the selection algorithm works and it is fair is that of a priority queue. The validators move ahead in this queue according to their voting power (the higher the voting power the faster a validator moves towards the head of the queue). When the algorithm runs the following happens: + +- all validators move "ahead" according to their powers: for each validator, increase the priority by the voting power +- first in the queue becomes the proposer: select the validator with highest priority +- move the proposer back in the queue: decrease the proposer's priority by the total voting power + +Notation: + +- vset - the validator set +- n - the number of validators +- VP(i) - voting power of validator i +- A(i) - accumulated priority for validator i +- P - total voting power of set +- avg - average of all validator priorities +- prop - proposer + +Simple view at the Selection Algorithm: + +```md + def ProposerSelection (vset): + + // compute priorities and elect proposer + for each validator i in vset: + A(i) += VP(i) + prop = max(A) + A(prop) -= P +``` + +## Stable Set + +Consider the validator set: + +Validator | p1 | p2 +----------|----|--- +VP | 1 | 3 + +Assuming no validator changes, the following table shows the proposer priority computation over a few runs. Four runs of the selection procedure are shown, starting with the 5th the same values are computed. +Each row shows the priority queue and the process place in it. The proposer is the closest to the head, the rightmost validator. As priorities are updated, the validators move right in the queue. The proposer moves left as its priority is reduced after election. + +| Priority Run | -2 | -1 | 0 | 1 | 2 | 3 | 4 | 5 | Alg step | +|----------------|----|----|-------|----|-------|----|----|----|------------------| +| | | | p1,p2 | | | | | | Initialized to 0 | +| run 1 | | | | p1 | | p2 | | | A(i)+=VP(i) | +| | | p2 | | p1 | | | | | A(p2)-= P | +| run 2 | | | | | p1,p2 | | | | A(i)+=VP(i) | +| | p1 | | | | p2 | | | | A(p1)-= P | +| run 3 | | p1 | | | | | | p2 | A(i)+=VP(i) | +| | | p1 | | p2 | | | | | A(p2)-= P | +| run 4 | | | p1 | | | | p2 | | A(i)+=VP(i) | +| | | | p1,p2 | | | | | | A(p2)-= P | + +It can be shown that: + +- At the end of each run k+1 the sum of the priorities is the same as at end of run k. If a new set's priorities are initialized to 0 then the sum of priorities will be 0 at each run while there are no changes. +- The max distance between priorites is (n-1) *P.*[formal proof not finished]* + +## Validator Set Changes + +Between proposer selection runs the validator set may change. Some changes have implications on the proposer election. + +### Voting Power Change + +Consider again the earlier example and assume that the voting power of p1 is changed to 4: + +Validator | p1 | p2 +----------|----|--- +VP | 4 | 3 + +Let's also assume that before this change the proposer priorites were as shown in first row (last run). As it can be seen, the selection could run again, without changes, as before. + +| Priority Run | -2 | -1 | 0 | 1 | 2 | Comment | +|----------------|----|----|---|----|----|-------------------| +| last run | | p2 | | p1 | | __update VP(p1)__ | +| next run | | | | | p2 | A(i)+=VP(i) | +| | p1 | | | | p2 | A(p1)-= P | + +However, when a validator changes power from a high to a low value, some other validator remain far back in the queue for a long time. This scenario is considered again in the Proposer Priority Range section. + +As before: + +- At the end of each run k+1 the sum of the priorities is the same as at run k. +- The max distance between priorites is (n-1) * P. + +### Validator Removal + +Consider a new example with set: + +Validator | p1 | p2 | p3 +----------|----|----|--- +VP | 1 | 2 | 3 + +Let's assume that after the last run the proposer priorities were as shown in first row with their sum being 0. After p2 is removed, at the end of next proposer selection run (penultimate row) the sum of priorities is -2 (minus the priority of the removed process). + +The procedure could continue without modifications. However, after a sufficiently large number of modifications in validator set, the priority values would migrate towards maximum or minimum allowed values causing truncations due to overflow detection. +For this reason, the selection procedure adds another __new step__ that centers the current priority values such that the priority sum remains close to 0. + +| Priority Run | -3 | -2 | -1 | 0 | 1 | 2 | 3 | Comment | +|----------------|----|----|----|---|----|----|----|-----------------------| +| last run | p3 | | | | p1 | p2 | | __remove p2__ | +| nextrun | | | | | | | | | +| __new step__ | | p3 | | | | p1 | | A(i) -= avg, avg = -1 | +| | | | | | p3 | | p1 | A(i)+=VP(i) | +| | | | p1 | | p3 | | | A(p1)-= P | + +The modified selection algorithm is: + +```md + def ProposerSelection (vset): + + // center priorities around zero + avg = sum(A(i) for i in vset)/len(vset) + for each validator i in vset: + A(i) -= avg + + // compute priorities and elect proposer + for each validator i in vset: + A(i) += VP(i) + prop = max(A) + A(prop) -= P +``` + +Observations: + +- The sum of priorities is now close to 0. Due to integer division the sum is an integer in (-n, n), where n is the number of validators. + +### New Validator + +When a new validator is added, same problem as the one described for removal appears, the sum of priorities in the new set is not zero. This is fixed with the centering step introduced above. + +One other issue that needs to be addressed is the following. A validator V that has just been elected is moved to the end of the queue. If the validator set is large and/ or other validators have significantly higher power, V will have to wait many runs to be elected. If V removes and re-adds itself to the set, it would make a significant (albeit unfair) "jump" ahead in the queue. + +In order to prevent this, when a new validator is added, its initial priority is set to: + +```md + A(V) = -1.125 * P +``` + +where P is the total voting power of the set including V. + +Current implementation uses the penalty factor of 1.125 because it provides a small punishment that is efficient to calculate. See [here](https://github.com/tendermint/tendermint/pull/2785#discussion_r235038971) for more details. + +If we consider the validator set where p3 has just been added: + +Validator | p1 | p2 | p3 +----------|----|----|--- +VP | 1 | 3 | 8 + +then p3 will start with proposer priority: + +```md + A(p3) = -1.125 * (1 + 3 + 8) ~ -13 +``` + +Note that since current computation uses integer division there is penalty loss when sum of the voting power is less than 8. + +In the next run, p3 will still be ahead in the queue, elected as proposer and moved back in the queue. + +| Priority Run | -13 | -9 | -5 | -2 | -1 | 0 | 1 | 2 | 5 | 6 | 7 | Alg step | +|----------------|-----|----|----|----|----|---|---|----|----|----|----|-----------------------| +| last run | | | | p2 | | | | p1 | | | | __add p3__ | +| | p3 | | | p2 | | | | p1 | | | | A(p3) = -13 | +| next run | | p3 | | | | | | p2 | | p1 | | A(i) -= avg, avg = -4 | +| | | | | | p3 | | | | p2 | | p1 | A(i)+=VP(i) | +| | | | p1 | | p3 | | | | p2 | | | A(p1)-=P | + +## Proposer Priority Range + +With the introduction of centering, some interesting cases occur. Low power validators that bind early in a set that includes high power validator(s) benefit from subsequent additions to the set. This is because these early validators run through more right shift operations during centering, operations that increase their priority. + +As an example, consider the set where p2 is added after p1, with priority -1.125 * 80k = -90k. After the selection procedure runs once: + +Validator | p1 | p2 | Comment +----------|------|------|------------------ +VP | 80k | 10 | +A | 0 | -90k | __added p2__ +A | 45k | -45k | __run selection__ + +Then execute the following steps: + +1. Add a new validator p3: + + Validator | p1 | p2 | p3 + ----------|-----|----|--- + VP | 80k | 10 | 10 + +2. Run selection once. The notation '..p'/'p..' means very small deviations compared to column priority. + + | Priority Run | -90k.. | -60k | -45k | -15k | 0 | 45k | 75k | 155k | Comment | + |---------------|--------|------|------|------|---|-----|-----|------|--------------| + | last run | p3 | | p2 | | | p1 | | | __added p3__ | + | next run + | *right_shift*| | p3 | | p2 | | | p1 | | A(i) -= avg,avg=-30k + | | | ..p3| | ..p2| | | | p1 | A(i)+=VP(i) + | | | ..p3| | ..p2| | | p1.. | | A(p1)-=P, P=80k+20 + +3. Remove p1 and run selection once: + + Validator | p3 | p2 | Comment + ----------|--------|-------|------------------ + VP | 10 | 10 | + A | -60k | -15k | + A | -22.5k | 22.5k | __run selection__ + +At this point, while the total voting power is 20, the distance between priorities is 45k. It will take 4500 runs for p3 to catch up with p2. + +In order to prevent these types of scenarios, the selection algorithm performs scaling of priorities such that the difference between min and max values is smaller than two times the total voting power. + +The modified selection algorithm is: + +```md + def ProposerSelection (vset): + + // scale the priority values + diff = max(A)-min(A) + threshold = 2 * P + if diff > threshold: + scale = diff/threshold + for each validator i in vset: + A(i) = A(i)/scale + + // center priorities around zero + avg = sum(A(i) for i in vset)/len(vset) + for each validator i in vset: + A(i) -= avg + + // compute priorities and elect proposer + for each validator i in vset: + A(i) += VP(i) + prop = max(A) + A(prop) -= P +``` + +Observations: + +- With this modification, the maximum distance between priorites becomes 2 * P. + +Note also that even during steady state the priority range may increase beyond 2 * P. The scaling introduced here helps to keep the range bounded. + +## Wrinkles + +### Validator Power Overflow Conditions + +The validator voting power is a positive number stored as an int64. When a validator is added the `1.125 * P` computation must not overflow. As a consequence the code handling validator updates (add and update) checks for overflow conditions making sure the total voting power is never larger than the largest int64 `MAX`, with the property that `1.125 * MAX` is still in the bounds of int64. Fatal error is return when overflow condition is detected. + +### Proposer Priority Overflow/ Underflow Handling + +The proposer priority is stored as an int64. The selection algorithm performs additions and subtractions to these values and in the case of overflows and underflows it limits the values to: + +```go + MaxInt64 = 1 << 63 - 1 + MinInt64 = -1 << 63 +``` + +## Requirement Fulfillment Claims + +__[R1]__ + +The proposer algorithm is deterministic giving consistent results across executions with same transactions and validator set modifications. +[WIP - needs more detail] + +__[R2]__ + +Given a set of processes with the total voting power P, during a sequence of elections of length P, the number of times any process is selected as proposer is equal to its voting power. The sequence of the P proposers then repeats. If we consider the validator set: + +Validator | p1 | p2 +----------|----|--- +VP | 1 | 3 + +With no other changes to the validator set, the current implementation of proposer selection generates the sequence: +`p2, p1, p2, p2, p2, p1, p2, p2,...` or [`p2, p1, p2, p2`]* +A sequence that starts with any circular permutation of the [`p2, p1, p2, p2`] sub-sequence would also provide the same degree of fairness. In fact these circular permutations show in the sliding window (over the generated sequence) of size equal to the length of the sub-sequence. + +Assigning priorities to each validator based on the voting power and updating them at each run ensures the fairness of the proposer selection. In addition, every time a validator is elected as proposer its priority is decreased with the total voting power. + +Intuitively, a process v jumps ahead in the queue at most (max(A) - min(A))/VP(v) times until it reaches the head and is elected. The frequency is then: + +```md + f(v) ~ VP(v)/(max(A)-min(A)) = 1/k * VP(v)/P +``` + +For current implementation, this means v should be proposer at least VP(v) times out of k * P runs, with scaling factor k=2. diff --git a/cometbft/v0.38/spec/consensus/Validator-Signing.mdx b/cometbft/v0.38/spec/consensus/Validator-Signing.mdx new file mode 100644 index 00000000..68547eea --- /dev/null +++ b/cometbft/v0.38/spec/consensus/Validator-Signing.mdx @@ -0,0 +1,233 @@ +--- +order: 5 +--- + +# Validator Signing + +Here we specify the rules for validating a proposal and vote before signing. +First we include some general notes on validating data structures common to both types. +We then provide specific validation rules for each. Finally, we include validation rules to prevent double-sigining. + +## SignedMsgType + +The `SignedMsgType` is a single byte that refers to the type of the message +being signed. It is defined in Go as follows: + +```go +// SignedMsgType is a type of signed message in the consensus. +type SignedMsgType byte + +const ( + // Votes + PrevoteType SignedMsgType = 0x01 + PrecommitType SignedMsgType = 0x02 + + // Proposals + ProposalType SignedMsgType = 0x20 +) +``` + +All signed messages must correspond to one of these types. + +## Timestamp + +Timestamp validation is subtle and there are currently no bounds placed on the +timestamp included in a proposal or vote. It is expected that validators will honestly +report their local clock time. The median of all timestamps +included in a commit is used as the timestamp for the next block height. + +Timestamps are expected to be strictly monotonic for a given validator, though +this is not currently enforced. + +## ChainID + +ChainID is an unstructured string with a max length of 50-bytes. +In the future, the ChainID may become structured, and may take on longer lengths. +For now, it is recommended that signers be configured for a particular ChainID, +and to only sign votes and proposals corresponding to that ChainID. + +## BlockID + +BlockID is the structure used to represent the block: + +```go +type BlockID struct { + Hash []byte + PartsHeader PartSetHeader +} + +type PartSetHeader struct { + Hash []byte + Total int +} +``` + +To be included in a valid vote or proposal, BlockID must either represent a `nil` block, or a complete one. +We introduce two methods, `BlockID.IsZero()` and `BlockID.IsComplete()` for these cases, respectively. + +`BlockID.IsZero()` returns true for BlockID `b` if each of the following +are true: + +```go +b.Hash == nil +b.PartsHeader.Total == 0 +b.PartsHeader.Hash == nil +``` + +`BlockID.IsComplete()` returns true for BlockID `b` if each of the following +are true: + +```go +len(b.Hash) == 32 +b.PartsHeader.Total > 0 +len(b.PartsHeader.Hash) == 32 +``` + +## Proposals + +The structure of a proposal for signing looks like: + +```go +type CanonicalProposal struct { + Type SignedMsgType // type alias for byte + Height int64 `binary:"fixed64"` + Round int64 `binary:"fixed64"` + POLRound int64 `binary:"fixed64"` + BlockID BlockID + Timestamp time.Time + ChainID string +} +``` + +A proposal is valid if each of the following lines evaluates to true for proposal `p`: + +```go +p.Type == 0x20 +p.Height > 0 +p.Round >= 0 +p.POLRound >= -1 +p.BlockID.IsComplete() +``` + +In other words, a proposal is valid for signing if it contains the type of a Proposal +(0x20), has a positive, non-zero height, a +non-negative round, a POLRound not less than -1, and a complete BlockID. + +## Votes + +The structure of a vote for signing looks like: + +```go +type CanonicalVote struct { + Type SignedMsgType // type alias for byte + Height int64 `binary:"fixed64"` + Round int64 `binary:"fixed64"` + BlockID BlockID + Timestamp time.Time + ChainID string +} +``` + +A vote is valid if each of the following lines evaluates to true for vote `v`: + +```go +v.Type == 0x1 || v.Type == 0x2 +v.Height > 0 +v.Round >= 0 +v.BlockID.IsZero() || v.BlockID.IsComplete() +``` + +In other words, a vote is valid for signing if it contains the type of a Prevote +or Precommit (0x1 or 0x2, respectively), has a positive, non-zero height, a +non-negative round, and an empty or valid BlockID. + +## Invalid Votes and Proposals + +Votes and proposals which do not satisfy the above rules are considered invalid. +Peers gossipping invalid votes and proposals may be disconnected from other peers on the network. +Note, however, that there is not currently any explicit mechanism to punish validators signing votes or proposals that fail +these basic validation rules. + +## Double Signing + +Signers must be careful not to sign conflicting messages, also known as "double signing" or "equivocating". +CometBFT has mechanisms to publish evidence of validators that signed conflicting votes, so they can be punished +by the application. Note CometBFT does not currently handle evidence of conflciting proposals, though it may in the future. + +### State + +To prevent such double signing, signers must track the height, round, and type of the last message signed. +Assume the signer keeps the following state, `s`: + +```go +type LastSigned struct { + Height int64 + Round int64 + Type SignedMsgType // byte +} +``` + +After signing a vote or proposal `m`, the signer sets: + +```go +s.Height = m.Height +s.Round = m.Round +s.Type = m.Type +``` + +### Proposals + +A signer should only sign a proposal `p` if any of the following lines are true: + +```go +p.Height > s.Height +p.Height == s.Height && p.Round > s.Round +``` + +In other words, a proposal should only be signed if it's at a higher height, or a higher round for the same height. +Once a proposal or vote has been signed for a given height and round, a proposal should never be signed for the same height and round. + +### Votes + +A signer should only sign a vote `v` if any of the following lines are true: + +```go +v.Height > s.Height +v.Height == s.Height && v.Round > s.Round +v.Height == s.Height && v.Round == s.Round && v.Step == 0x1 && s.Step == 0x20 +v.Height == s.Height && v.Round == s.Round && v.Step == 0x2 && s.Step != 0x2 +``` + +In other words, a vote should only be signed if it's: + +- at a higher height +- at a higher round for the same height +- a prevote for the same height and round where we haven't signed a prevote or precommit (but have signed a proposal) +- a precommit for the same height and round where we haven't signed a precommit (but have signed a proposal and/or a prevote) + +This means that once a validator signs a prevote for a given height and round, the only other message it can sign for that height and round is a precommit. +And once a validator signs a precommit for a given height and round, it must not sign any other message for that same height and round. + +Note this includes votes for `nil`, ie. where `BlockID.IsZero()` is true. If a +signer has already signed a vote where `BlockID.IsZero()` is true, it cannot +sign another vote with the same type for the same height and round where +`BlockID.IsComplete()` is true. Thus only a single vote of a particular type +(ie. 0x01 or 0x02) can be signed for the same height and round. + +### Other Rules + +According to the rules of Tendermint consensus algorithm, adopted in CometBFT, once a validator precommits for +a block, they become "locked" on that block, which means they can't prevote for +another block unless they see sufficient justification (ie. a polka from a +higher round). For more details, see the [consensus +spec](https://arxiv.org/abs/1807.04938). + +Violating this rule is known as "amnesia". In contrast to equivocation, +which is easy to detect, amnesia is difficult to detect without access to votes +from all the validators, as this is what constitutes the justification for +"unlocking". Hence, amnesia is not punished within the protocol, and cannot +easily be prevented by a signer. If enough validators simultaneously commit an +amnesia attack, they may cause a fork of the blockchain, at which point an +off-chain protocol must be engaged to collect votes from all the validators and +determine who misbehaved. For more details, see [fork +detection](https://github.com/tendermint/tendermint/pull/3978). diff --git a/cometbft/v0.38/spec/consensus/WAL.mdx b/cometbft/v0.38/spec/consensus/WAL.mdx new file mode 100644 index 00000000..599d63d3 --- /dev/null +++ b/cometbft/v0.38/spec/consensus/WAL.mdx @@ -0,0 +1,35 @@ +--- +order: 6 +--- +# WAL + +Consensus module writes every message to the WAL (write-ahead log). + +It also issues fsync syscall through +[File#Sync](https://golang.org/pkg/os/#File.Sync) for messages signed by this +node (to prevent double signing). + +Under the hood, it uses +[autofile.Group](https://github.com/cometbft/cometbft/blob/af3bc47df982e271d4d340a3c5e0d773e440466d/libs/autofile/group.go#L54), +which rotates files when those get too big (> 10MB). + +The total maximum size is 1GB. We only need the latest block and the block before it, +but if the former is dragging on across many rounds, we want all those rounds. + +## Replay + +Consensus module will replay all the messages of the last height written to WAL +before a crash (if such occurs). + +The private validator may try to sign messages during replay because it runs +somewhat autonomously and does not know about replay process. + +For example, if we got all the way to precommit in the WAL and then crash, +after we replay the proposal message, the private validator will try to sign a +prevote. But it will fail. That's ok because we’ll see the prevote later in the +WAL. Then it will go to precommit, and that time it will work because the +private validator contains the `LastSignBytes` and then we’ll replay the +precommit from the WAL. + +Make sure to read about [WAL corruption](https://github.com/cometbft/cometbft/blob/v0.38.x/docs/core/running-in-production.md#wal-corruption) +and recovery strategies. diff --git a/cometbft/v0.38/spec/consensus/consensus-paper/IEEEtran.bst b/cometbft/v0.38/spec/consensus/consensus-paper/IEEEtran.bst new file mode 100644 index 00000000..53fbc030 --- /dev/null +++ b/cometbft/v0.38/spec/consensus/consensus-paper/IEEEtran.bst @@ -0,0 +1,2417 @@ +%% +%% IEEEtran.bst +%% BibTeX Bibliography Style file for IEEE Journals and Conferences (unsorted) +%% Version 1.12 (2007/01/11) +%% +%% Copyright (c) 2003-2007 Michael Shell +%% +%% Original starting code base and algorithms obtained from the output of +%% Patrick W. Daly's makebst package as well as from prior versions of +%% IEEE BibTeX styles: +%% +%% 1. Howard Trickey and Oren Patashnik's ieeetr.bst (1985/1988) +%% 2. Silvano Balemi and Richard H. Roy's IEEEbib.bst (1993) +%% +%% Support sites: +%% http://www.michaelshell.org/tex/ieeetran/ +%% http://www.ctan.org/tex-archive/macros/latex/contrib/IEEEtran/ +%% and/or +%% http://www.ieee.org/ +%% +%% For use with BibTeX version 0.99a or later +%% +%% This is a numerical citation style. +%% +%%************************************************************************* +%% Legal Notice: +%% This code is offered as-is without any warranty either expressed or +%% implied; without even the implied warranty of MERCHANTABILITY or +%% FITNESS FOR A PARTICULAR PURPOSE! +%% User assumes all risk. +%% In no event shall IEEE or any contributor to this code be liable for +%% any damages or losses, including, but not limited to, incidental, +%% consequential, or any other damages, resulting from the use or misuse +%% of any information contained here. +%% +%% All comments are the opinions of their respective authors and are not +%% necessarily endorsed by the IEEE. +%% +%% This work is distributed under the LaTeX Project Public License (LPPL) +%% ( http://www.latex-project.org/ ) version 1.3, and may be freely used, +%% distributed and modified. A copy of the LPPL, version 1.3, is included +%% in the base LaTeX documentation of all distributions of LaTeX released +%% 2003/12/01 or later. +%% Retain all contribution notices and credits. +%% ** Modified files should be clearly indicated as such, including ** +%% ** renaming them and changing author support contact information. ** +%% +%% File list of work: IEEEabrv.bib, IEEEfull.bib, IEEEexample.bib, +%% IEEEtran.bst, IEEEtranS.bst, IEEEtranSA.bst, +%% IEEEtranN.bst, IEEEtranSN.bst, IEEEtran_bst_HOWTO.pdf +%%************************************************************************* +% +% +% Changelog: +% +% 1.00 (2002/08/13) Initial release +% +% 1.10 (2002/09/27) +% 1. Corrected minor bug for improperly formed warning message when a +% book was not given a title. Thanks to Ming Kin Lai for reporting this. +% 2. Added support for CTLname_format_string and CTLname_latex_cmd fields +% in the BST control entry type. +% +% 1.11 (2003/04/02) +% 1. Fixed bug with URLs containing underscores when using url.sty. Thanks +% to Ming Kin Lai for reporting this. +% +% 1.12 (2007/01/11) +% 1. Fixed bug with unwanted comma before "et al." when an entry contained +% more than two author names. Thanks to Pallav Gupta for reporting this. +% 2. Fixed bug with anomalous closing quote in tech reports that have a +% type, but without a number or address. Thanks to Mehrdad Mirreza for +% reporting this. +% 3. Use braces in \providecommand in begin.bib to better support +% latex2html. TeX style length assignments OK with recent versions +% of latex2html - 1.71 (2002/2/1) or later is strongly recommended. +% Use of the language field still causes trouble with latex2html. +% Thanks to Federico Beffa for reporting this. +% 4. Added IEEEtran.bst ID and version comment string to .bbl output. +% 5. Provide a \BIBdecl hook that allows the user to execute commands +% just prior to the first entry. +% 6. Use default urlstyle (is using url.sty) of "same" rather than rm to +% better work with a wider variety of bibliography styles. +% 7. Changed month abbreviations from Sept., July and June to Sep., Jul., +% and Jun., respectively, as IEEE now does. Thanks to Moritz Borgmann +% for reporting this. +% 8. Control entry types should not be considered when calculating longest +% label width. +% 9. Added alias www for electronic/online. +% 10. Added CTLname_url_prefix control entry type. + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% DEFAULTS FOR THE CONTROLS OF THE BST STYLE %% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +% These are the defaults for the user adjustable controls. The values used +% here can be overridden by the user via IEEEtranBSTCTL entry type. + +% NOTE: The recommended LaTeX command to invoke a control entry type is: +% +%\makeatletter +%\def\bstctlcite{\@ifnextchar[{\@bstctlcite}{\@bstctlcite[@auxout]}} +%\def\@bstctlcite[#1]#2{\@bsphack +% \@for\@citeb:=#2\do{% +% \edef\@citeb{\expandafter\@firstofone\@citeb}% +% \if@filesw\immediate\write\csname #1\endcsname{\string\citation{\@citeb}}\fi}% +% \@esphack} +%\makeatother +% +% It is called at the start of the document, before the first \cite, like: +% \bstctlcite{IEEEexample:BSTcontrol} +% +% IEEEtran.cls V1.6 and later does provide this command. + + + +% #0 turns off the display of the number for articles. +% #1 enables +FUNCTION {default.is.use.number.for.article} { #1 } + + +% #0 turns off the display of the paper and type fields in @inproceedings. +% #1 enables +FUNCTION {default.is.use.paper} { #1 } + + +% #0 turns off the forced use of "et al." +% #1 enables +FUNCTION {default.is.forced.et.al} { #0 } + +% The maximum number of names that can be present beyond which an "et al." +% usage is forced. Be sure that num.names.shown.with.forced.et.al (below) +% is not greater than this value! +% Note: There are many instances of references in IEEE journals which have +% a very large number of authors as well as instances in which "et al." is +% used profusely. +FUNCTION {default.max.num.names.before.forced.et.al} { #10 } + +% The number of names that will be shown with a forced "et al.". +% Must be less than or equal to max.num.names.before.forced.et.al +FUNCTION {default.num.names.shown.with.forced.et.al} { #1 } + + +% #0 turns off the alternate interword spacing for entries with URLs. +% #1 enables +FUNCTION {default.is.use.alt.interword.spacing} { #1 } + +% If alternate interword spacing for entries with URLs is enabled, this is +% the interword spacing stretch factor that will be used. For example, the +% default "4" here means that the interword spacing in entries with URLs can +% stretch to four times normal. Does not have to be an integer. Note that +% the value specified here can be overridden by the user in their LaTeX +% code via a command such as: +% "\providecommand\BIBentryALTinterwordstretchfactor{1.5}" in addition to +% that via the IEEEtranBSTCTL entry type. +FUNCTION {default.ALTinterwordstretchfactor} { "4" } + + +% #0 turns off the "dashification" of repeated (i.e., identical to those +% of the previous entry) names. IEEE normally does this. +% #1 enables +FUNCTION {default.is.dash.repeated.names} { #1 } + + +% The default name format control string. +FUNCTION {default.name.format.string}{ "{f.~}{vv~}{ll}{, jj}" } + + +% The default LaTeX font command for the names. +FUNCTION {default.name.latex.cmd}{ "" } + + +% The default URL prefix. +FUNCTION {default.name.url.prefix}{ "[Online]. Available:" } + + +% Other controls that cannot be accessed via IEEEtranBSTCTL entry type. + +% #0 turns off the terminal startup banner/completed message so as to +% operate more quietly. +% #1 enables +FUNCTION {is.print.banners.to.terminal} { #1 } + + + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% FILE VERSION AND BANNER %% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +FUNCTION{bst.file.version} { "1.12" } +FUNCTION{bst.file.date} { "2007/01/11" } +FUNCTION{bst.file.website} { "http://www.michaelshell.org/tex/ieeetran/bibtex/" } + +FUNCTION {banner.message} +{ is.print.banners.to.terminal + { "-- IEEEtran.bst version" " " * bst.file.version * + " (" * bst.file.date * ") " * "by Michael Shell." * + top$ + "-- " bst.file.website * + top$ + "-- See the " quote$ * "IEEEtran_bst_HOWTO.pdf" * quote$ * " manual for usage information." * + top$ + } + { skip$ } + if$ +} + +FUNCTION {completed.message} +{ is.print.banners.to.terminal + { "" + top$ + "Done." + top$ + } + { skip$ } + if$ +} + + + + +%%%%%%%%%%%%%%%%%%%%%% +%% STRING CONSTANTS %% +%%%%%%%%%%%%%%%%%%%%%% + +FUNCTION {bbl.and}{ "and" } +FUNCTION {bbl.etal}{ "et~al." } +FUNCTION {bbl.editors}{ "eds." } +FUNCTION {bbl.editor}{ "ed." } +FUNCTION {bbl.edition}{ "ed." } +FUNCTION {bbl.volume}{ "vol." } +FUNCTION {bbl.of}{ "of" } +FUNCTION {bbl.number}{ "no." } +FUNCTION {bbl.in}{ "in" } +FUNCTION {bbl.pages}{ "pp." } +FUNCTION {bbl.page}{ "p." } +FUNCTION {bbl.chapter}{ "ch." } +FUNCTION {bbl.paper}{ "paper" } +FUNCTION {bbl.part}{ "pt." } +FUNCTION {bbl.patent}{ "Patent" } +FUNCTION {bbl.patentUS}{ "U.S." } +FUNCTION {bbl.revision}{ "Rev." } +FUNCTION {bbl.series}{ "ser." } +FUNCTION {bbl.standard}{ "Std." } +FUNCTION {bbl.techrep}{ "Tech. Rep." } +FUNCTION {bbl.mthesis}{ "Master's thesis" } +FUNCTION {bbl.phdthesis}{ "Ph.D. dissertation" } +FUNCTION {bbl.st}{ "st" } +FUNCTION {bbl.nd}{ "nd" } +FUNCTION {bbl.rd}{ "rd" } +FUNCTION {bbl.th}{ "th" } + + +% This is the LaTeX spacer that is used when a larger than normal space +% is called for (such as just before the address:publisher). +FUNCTION {large.space} { "\hskip 1em plus 0.5em minus 0.4em\relax " } + +% The LaTeX code for dashes that are used to represent repeated names. +% Note: Some older IEEE journals used something like +% "\rule{0.275in}{0.5pt}\," which is fairly thick and runs right along +% the baseline. However, IEEE now uses a thinner, above baseline, +% six dash long sequence. +FUNCTION {repeated.name.dashes} { "------" } + + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% PREDEFINED STRING MACROS %% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +MACRO {jan} {"Jan."} +MACRO {feb} {"Feb."} +MACRO {mar} {"Mar."} +MACRO {apr} {"Apr."} +MACRO {may} {"May"} +MACRO {jun} {"Jun."} +MACRO {jul} {"Jul."} +MACRO {aug} {"Aug."} +MACRO {sep} {"Sep."} +MACRO {oct} {"Oct."} +MACRO {nov} {"Nov."} +MACRO {dec} {"Dec."} + + + +%%%%%%%%%%%%%%%%%% +%% ENTRY FIELDS %% +%%%%%%%%%%%%%%%%%% + +ENTRY + { address + assignee + author + booktitle + chapter + day + dayfiled + edition + editor + howpublished + institution + intype + journal + key + language + month + monthfiled + nationality + note + number + organization + pages + paper + publisher + school + series + revision + title + type + url + volume + year + yearfiled + CTLuse_article_number + CTLuse_paper + CTLuse_forced_etal + CTLmax_names_forced_etal + CTLnames_show_etal + CTLuse_alt_spacing + CTLalt_stretch_factor + CTLdash_repeated_names + CTLname_format_string + CTLname_latex_cmd + CTLname_url_prefix + } + {} + { label } + + + + +%%%%%%%%%%%%%%%%%%%%%%% +%% INTEGER VARIABLES %% +%%%%%%%%%%%%%%%%%%%%%%% + +INTEGERS { prev.status.punct this.status.punct punct.std + punct.no punct.comma punct.period + prev.status.space this.status.space space.std + space.no space.normal space.large + prev.status.quote this.status.quote quote.std + quote.no quote.close + prev.status.nline this.status.nline nline.std + nline.no nline.newblock + status.cap cap.std + cap.no cap.yes} + +INTEGERS { longest.label.width multiresult nameptr namesleft number.label numnames } + +INTEGERS { is.use.number.for.article + is.use.paper + is.forced.et.al + max.num.names.before.forced.et.al + num.names.shown.with.forced.et.al + is.use.alt.interword.spacing + is.dash.repeated.names} + + +%%%%%%%%%%%%%%%%%%%%%% +%% STRING VARIABLES %% +%%%%%%%%%%%%%%%%%%%%%% + +STRINGS { bibinfo + longest.label + oldname + s + t + ALTinterwordstretchfactor + name.format.string + name.latex.cmd + name.url.prefix} + + + + +%%%%%%%%%%%%%%%%%%%%%%%%% +%% LOW LEVEL FUNCTIONS %% +%%%%%%%%%%%%%%%%%%%%%%%%% + +FUNCTION {initialize.controls} +{ default.is.use.number.for.article 'is.use.number.for.article := + default.is.use.paper 'is.use.paper := + default.is.forced.et.al 'is.forced.et.al := + default.max.num.names.before.forced.et.al 'max.num.names.before.forced.et.al := + default.num.names.shown.with.forced.et.al 'num.names.shown.with.forced.et.al := + default.is.use.alt.interword.spacing 'is.use.alt.interword.spacing := + default.is.dash.repeated.names 'is.dash.repeated.names := + default.ALTinterwordstretchfactor 'ALTinterwordstretchfactor := + default.name.format.string 'name.format.string := + default.name.latex.cmd 'name.latex.cmd := + default.name.url.prefix 'name.url.prefix := +} + + +% This IEEEtran.bst features a very powerful and flexible mechanism for +% controlling the capitalization, punctuation, spacing, quotation, and +% newlines of the formatted entry fields. (Note: IEEEtran.bst does not need +% or use the newline/newblock feature, but it has been implemented for +% possible future use.) The output states of IEEEtran.bst consist of +% multiple independent attributes and, as such, can be thought of as being +% vectors, rather than the simple scalar values ("before.all", +% "mid.sentence", etc.) used in most other .bst files. +% +% The more flexible and complex design used here was motivated in part by +% IEEE's rather unusual bibliography style. For example, IEEE ends the +% previous field item with a period and large space prior to the publisher +% address; the @electronic entry types use periods as inter-item punctuation +% rather than the commas used by the other entry types; and URLs are never +% followed by periods even though they are the last item in the entry. +% Although it is possible to accommodate these features with the conventional +% output state system, the seemingly endless exceptions make for convoluted, +% unreliable and difficult to maintain code. +% +% IEEEtran.bst's output state system can be easily understood via a simple +% illustration of two most recently formatted entry fields (on the stack): +% +% CURRENT_ITEM +% "PREVIOUS_ITEM +% +% which, in this example, is to eventually appear in the bibliography as: +% +% "PREVIOUS_ITEM," CURRENT_ITEM +% +% It is the job of the output routine to take the previous item off of the +% stack (while leaving the current item at the top of the stack), apply its +% trailing punctuation (including closing quote marks) and spacing, and then +% to write the result to BibTeX's output buffer: +% +% "PREVIOUS_ITEM," +% +% Punctuation (and spacing) between items is often determined by both of the +% items rather than just the first one. The presence of quotation marks +% further complicates the situation because, in standard English, trailing +% punctuation marks are supposed to be contained within the quotes. +% +% IEEEtran.bst maintains two output state (aka "status") vectors which +% correspond to the previous and current (aka "this") items. Each vector +% consists of several independent attributes which track punctuation, +% spacing, quotation, and newlines. Capitalization status is handled by a +% separate scalar because the format routines, not the output routine, +% handle capitalization and, therefore, there is no need to maintain the +% capitalization attribute for both the "previous" and "this" items. +% +% When a format routine adds a new item, it copies the current output status +% vector to the previous output status vector and (usually) resets the +% current (this) output status vector to a "standard status" vector. Using a +% "standard status" vector in this way allows us to redefine what we mean by +% "standard status" at the start of each entry handler and reuse the same +% format routines under the various inter-item separation schemes. For +% example, the standard status vector for the @book entry type may use +% commas for item separators, while the @electronic type may use periods, +% yet both entry handlers exploit many of the exact same format routines. +% +% Because format routines have write access to the output status vector of +% the previous item, they can override the punctuation choices of the +% previous format routine! Therefore, it becomes trivial to implement rules +% such as "Always use a period and a large space before the publisher." By +% pushing the generation of the closing quote mark to the output routine, we +% avoid all the problems caused by having to close a quote before having all +% the information required to determine what the punctuation should be. +% +% The IEEEtran.bst output state system can easily be expanded if needed. +% For instance, it is easy to add a "space.tie" attribute value if the +% bibliography rules mandate that two items have to be joined with an +% unbreakable space. + +FUNCTION {initialize.status.constants} +{ #0 'punct.no := + #1 'punct.comma := + #2 'punct.period := + #0 'space.no := + #1 'space.normal := + #2 'space.large := + #0 'quote.no := + #1 'quote.close := + #0 'cap.no := + #1 'cap.yes := + #0 'nline.no := + #1 'nline.newblock := +} + +FUNCTION {std.status.using.comma} +{ punct.comma 'punct.std := + space.normal 'space.std := + quote.no 'quote.std := + nline.no 'nline.std := + cap.no 'cap.std := +} + +FUNCTION {std.status.using.period} +{ punct.period 'punct.std := + space.normal 'space.std := + quote.no 'quote.std := + nline.no 'nline.std := + cap.yes 'cap.std := +} + +FUNCTION {initialize.prev.this.status} +{ punct.no 'prev.status.punct := + space.no 'prev.status.space := + quote.no 'prev.status.quote := + nline.no 'prev.status.nline := + punct.no 'this.status.punct := + space.no 'this.status.space := + quote.no 'this.status.quote := + nline.no 'this.status.nline := + cap.yes 'status.cap := +} + +FUNCTION {this.status.std} +{ punct.std 'this.status.punct := + space.std 'this.status.space := + quote.std 'this.status.quote := + nline.std 'this.status.nline := +} + +FUNCTION {cap.status.std}{ cap.std 'status.cap := } + +FUNCTION {this.to.prev.status} +{ this.status.punct 'prev.status.punct := + this.status.space 'prev.status.space := + this.status.quote 'prev.status.quote := + this.status.nline 'prev.status.nline := +} + + +FUNCTION {not} +{ { #0 } + { #1 } + if$ +} + +FUNCTION {and} +{ { skip$ } + { pop$ #0 } + if$ +} + +FUNCTION {or} +{ { pop$ #1 } + { skip$ } + if$ +} + + +% convert the strings "yes" or "no" to #1 or #0 respectively +FUNCTION {yes.no.to.int} +{ "l" change.case$ duplicate$ + "yes" = + { pop$ #1 } + { duplicate$ "no" = + { pop$ #0 } + { "unknown boolean " quote$ * swap$ * quote$ * + " in " * cite$ * warning$ + #0 + } + if$ + } + if$ +} + + +% pushes true if the single char string on the stack is in the +% range of "0" to "9" +FUNCTION {is.num} +{ chr.to.int$ + duplicate$ "0" chr.to.int$ < not + swap$ "9" chr.to.int$ > not and +} + +% multiplies the integer on the stack by a factor of 10 +FUNCTION {bump.int.mag} +{ #0 'multiresult := + { duplicate$ #0 > } + { #1 - + multiresult #10 + + 'multiresult := + } + while$ +pop$ +multiresult +} + +% converts a single character string on the stack to an integer +FUNCTION {char.to.integer} +{ duplicate$ + is.num + { chr.to.int$ "0" chr.to.int$ - } + {"noninteger character " quote$ * swap$ * quote$ * + " in integer field of " * cite$ * warning$ + #0 + } + if$ +} + +% converts a string on the stack to an integer +FUNCTION {string.to.integer} +{ duplicate$ text.length$ 'namesleft := + #1 'nameptr := + #0 'numnames := + { nameptr namesleft > not } + { duplicate$ nameptr #1 substring$ + char.to.integer numnames bump.int.mag + + 'numnames := + nameptr #1 + + 'nameptr := + } + while$ +pop$ +numnames +} + + + + +% The output routines write out the *next* to the top (previous) item on the +% stack, adding punctuation and such as needed. Since IEEEtran.bst maintains +% the output status for the top two items on the stack, these output +% routines have to consider the previous output status (which corresponds to +% the item that is being output). Full independent control of punctuation, +% closing quote marks, spacing, and newblock is provided. +% +% "output.nonnull" does not check for the presence of a previous empty +% item. +% +% "output" does check for the presence of a previous empty item and will +% remove an empty item rather than outputing it. +% +% "output.warn" is like "output", but will issue a warning if it detects +% an empty item. + +FUNCTION {output.nonnull} +{ swap$ + prev.status.punct punct.comma = + { "," * } + { skip$ } + if$ + prev.status.punct punct.period = + { add.period$ } + { skip$ } + if$ + prev.status.quote quote.close = + { "''" * } + { skip$ } + if$ + prev.status.space space.normal = + { " " * } + { skip$ } + if$ + prev.status.space space.large = + { large.space * } + { skip$ } + if$ + write$ + prev.status.nline nline.newblock = + { newline$ "\newblock " write$ } + { skip$ } + if$ +} + +FUNCTION {output} +{ duplicate$ empty$ + 'pop$ + 'output.nonnull + if$ +} + +FUNCTION {output.warn} +{ 't := + duplicate$ empty$ + { pop$ "empty " t * " in " * cite$ * warning$ } + 'output.nonnull + if$ +} + +% "fin.entry" is the output routine that handles the last item of the entry +% (which will be on the top of the stack when "fin.entry" is called). + +FUNCTION {fin.entry} +{ this.status.punct punct.no = + { skip$ } + { add.period$ } + if$ + this.status.quote quote.close = + { "''" * } + { skip$ } + if$ +write$ +newline$ +} + + +FUNCTION {is.last.char.not.punct} +{ duplicate$ + "}" * add.period$ + #-1 #1 substring$ "." = +} + +FUNCTION {is.multiple.pages} +{ 't := + #0 'multiresult := + { multiresult not + t empty$ not + and + } + { t #1 #1 substring$ + duplicate$ "-" = + swap$ duplicate$ "," = + swap$ "+" = + or or + { #1 'multiresult := } + { t #2 global.max$ substring$ 't := } + if$ + } + while$ + multiresult +} + +FUNCTION {capitalize}{ "u" change.case$ "t" change.case$ } + +FUNCTION {emphasize} +{ duplicate$ empty$ + { pop$ "" } + { "\emph{" swap$ * "}" * } + if$ +} + +FUNCTION {do.name.latex.cmd} +{ name.latex.cmd + empty$ + { skip$ } + { name.latex.cmd "{" * swap$ * "}" * } + if$ +} + +% IEEEtran.bst uses its own \BIBforeignlanguage command which directly +% invokes the TeX hyphenation patterns without the need of the Babel +% package. Babel does a lot more than switch hyphenation patterns and +% its loading can cause unintended effects in many class files (such as +% IEEEtran.cls). +FUNCTION {select.language} +{ duplicate$ empty$ 'pop$ + { language empty$ 'skip$ + { "\BIBforeignlanguage{" language * "}{" * swap$ * "}" * } + if$ + } + if$ +} + +FUNCTION {tie.or.space.prefix} +{ duplicate$ text.length$ #3 < + { "~" } + { " " } + if$ + swap$ +} + +FUNCTION {get.bbl.editor} +{ editor num.names$ #1 > 'bbl.editors 'bbl.editor if$ } + +FUNCTION {space.word}{ " " swap$ * " " * } + + +% Field Conditioners, Converters, Checkers and External Interfaces + +FUNCTION {empty.field.to.null.string} +{ duplicate$ empty$ + { pop$ "" } + { skip$ } + if$ +} + +FUNCTION {either.or.check} +{ empty$ + { pop$ } + { "can't use both " swap$ * " fields in " * cite$ * warning$ } + if$ +} + +FUNCTION {empty.entry.warn} +{ author empty$ title empty$ howpublished empty$ + month empty$ year empty$ note empty$ url empty$ + and and and and and and + { "all relevant fields are empty in " cite$ * warning$ } + 'skip$ + if$ +} + + +% The bibinfo system provides a way for the electronic parsing/acquisition +% of a bibliography's contents as is done by ReVTeX. For example, a field +% could be entered into the bibliography as: +% \bibinfo{volume}{2} +% Only the "2" would show up in the document, but the LaTeX \bibinfo command +% could do additional things with the information. IEEEtran.bst does provide +% a \bibinfo command via "\providecommand{\bibinfo}[2]{#2}". However, it is +% currently not used as the bogus bibinfo functions defined here output the +% entry values directly without the \bibinfo wrapper. The bibinfo functions +% themselves (and the calls to them) are retained for possible future use. +% +% bibinfo.check avoids acting on missing fields while bibinfo.warn will +% issue a warning message if a missing field is detected. Prior to calling +% the bibinfo functions, the user should push the field value and then its +% name string, in that order. + +FUNCTION {bibinfo.check} +{ swap$ duplicate$ missing$ + { pop$ pop$ "" } + { duplicate$ empty$ + { swap$ pop$ } + { swap$ pop$ } + if$ + } + if$ +} + +FUNCTION {bibinfo.warn} +{ swap$ duplicate$ missing$ + { swap$ "missing " swap$ * " in " * cite$ * warning$ pop$ "" } + { duplicate$ empty$ + { swap$ "empty " swap$ * " in " * cite$ * warning$ } + { swap$ pop$ } + if$ + } + if$ +} + + +% IEEE separates large numbers with more than 4 digits into groups of +% three. IEEE uses a small space to separate these number groups. +% Typical applications include patent and page numbers. + +% number of consecutive digits required to trigger the group separation. +FUNCTION {large.number.trigger}{ #5 } + +% For numbers longer than the trigger, this is the blocksize of the groups. +% The blocksize must be less than the trigger threshold, and 2 * blocksize +% must be greater than the trigger threshold (can't do more than one +% separation on the initial trigger). +FUNCTION {large.number.blocksize}{ #3 } + +% What is actually inserted between the number groups. +FUNCTION {large.number.separator}{ "\," } + +% So as to save on integer variables by reusing existing ones, numnames +% holds the current number of consecutive digits read and nameptr holds +% the number that will trigger an inserted space. +FUNCTION {large.number.separate} +{ 't := + "" + #0 'numnames := + large.number.trigger 'nameptr := + { t empty$ not } + { t #-1 #1 substring$ is.num + { numnames #1 + 'numnames := } + { #0 'numnames := + large.number.trigger 'nameptr := + } + if$ + t #-1 #1 substring$ swap$ * + t #-2 global.max$ substring$ 't := + numnames nameptr = + { duplicate$ #1 nameptr large.number.blocksize - substring$ swap$ + nameptr large.number.blocksize - #1 + global.max$ substring$ + large.number.separator swap$ * * + nameptr large.number.blocksize - 'numnames := + large.number.blocksize #1 + 'nameptr := + } + { skip$ } + if$ + } + while$ +} + +% Converts all single dashes "-" to double dashes "--". +FUNCTION {n.dashify} +{ large.number.separate + 't := + "" + { t empty$ not } + { t #1 #1 substring$ "-" = + { t #1 #2 substring$ "--" = not + { "--" * + t #2 global.max$ substring$ 't := + } + { { t #1 #1 substring$ "-" = } + { "-" * + t #2 global.max$ substring$ 't := + } + while$ + } + if$ + } + { t #1 #1 substring$ * + t #2 global.max$ substring$ 't := + } + if$ + } + while$ +} + + +% This function detects entries with names that are identical to that of +% the previous entry and replaces the repeated names with dashes (if the +% "is.dash.repeated.names" user control is nonzero). +FUNCTION {name.or.dash} +{ 's := + oldname empty$ + { s 'oldname := s } + { s oldname = + { is.dash.repeated.names + { repeated.name.dashes } + { s 'oldname := s } + if$ + } + { s 'oldname := s } + if$ + } + if$ +} + +% Converts the number string on the top of the stack to +% "numerical ordinal form" (e.g., "7" to "7th"). There is +% no artificial limit to the upper bound of the numbers as the +% least significant digit always determines the ordinal form. +FUNCTION {num.to.ordinal} +{ duplicate$ #-1 #1 substring$ "1" = + { bbl.st * } + { duplicate$ #-1 #1 substring$ "2" = + { bbl.nd * } + { duplicate$ #-1 #1 substring$ "3" = + { bbl.rd * } + { bbl.th * } + if$ + } + if$ + } + if$ +} + +% If the string on the top of the stack begins with a number, +% (e.g., 11th) then replace the string with the leading number +% it contains. Otherwise retain the string as-is. s holds the +% extracted number, t holds the part of the string that remains +% to be scanned. +FUNCTION {extract.num} +{ duplicate$ 't := + "" 's := + { t empty$ not } + { t #1 #1 substring$ + t #2 global.max$ substring$ 't := + duplicate$ is.num + { s swap$ * 's := } + { pop$ "" 't := } + if$ + } + while$ + s empty$ + 'skip$ + { pop$ s } + if$ +} + +% Converts the word number string on the top of the stack to +% Arabic string form. Will be successful up to "tenth". +FUNCTION {word.to.num} +{ duplicate$ "l" change.case$ 's := + s "first" = + { pop$ "1" } + { skip$ } + if$ + s "second" = + { pop$ "2" } + { skip$ } + if$ + s "third" = + { pop$ "3" } + { skip$ } + if$ + s "fourth" = + { pop$ "4" } + { skip$ } + if$ + s "fifth" = + { pop$ "5" } + { skip$ } + if$ + s "sixth" = + { pop$ "6" } + { skip$ } + if$ + s "seventh" = + { pop$ "7" } + { skip$ } + if$ + s "eighth" = + { pop$ "8" } + { skip$ } + if$ + s "ninth" = + { pop$ "9" } + { skip$ } + if$ + s "tenth" = + { pop$ "10" } + { skip$ } + if$ +} + + +% Converts the string on the top of the stack to numerical +% ordinal (e.g., "11th") form. +FUNCTION {convert.edition} +{ duplicate$ empty$ 'skip$ + { duplicate$ #1 #1 substring$ is.num + { extract.num + num.to.ordinal + } + { word.to.num + duplicate$ #1 #1 substring$ is.num + { num.to.ordinal } + { "edition ordinal word " quote$ * edition * quote$ * + " may be too high (or improper) for conversion" * " in " * cite$ * warning$ + } + if$ + } + if$ + } + if$ +} + + + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% LATEX BIBLIOGRAPHY CODE %% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +FUNCTION {start.entry} +{ newline$ + "\bibitem{" write$ + cite$ write$ + "}" write$ + newline$ + "" + initialize.prev.this.status +} + +% Here we write out all the LaTeX code that we will need. The most involved +% code sequences are those that control the alternate interword spacing and +% foreign language hyphenation patterns. The heavy use of \providecommand +% gives users a way to override the defaults. Special thanks to Javier Bezos, +% Johannes Braams, Robin Fairbairns, Heiko Oberdiek, Donald Arseneau and all +% the other gurus on comp.text.tex for their help and advice on the topic of +% \selectlanguage, Babel and BibTeX. +FUNCTION {begin.bib} +{ "% Generated by IEEEtran.bst, version: " bst.file.version * " (" * bst.file.date * ")" * + write$ newline$ + preamble$ empty$ 'skip$ + { preamble$ write$ newline$ } + if$ + "\begin{thebibliography}{" longest.label * "}" * + write$ newline$ + "\providecommand{\url}[1]{#1}" + write$ newline$ + "\csname url@samestyle\endcsname" + write$ newline$ + "\providecommand{\newblock}{\relax}" + write$ newline$ + "\providecommand{\bibinfo}[2]{#2}" + write$ newline$ + "\providecommand{\BIBentrySTDinterwordspacing}{\spaceskip=0pt\relax}" + write$ newline$ + "\providecommand{\BIBentryALTinterwordstretchfactor}{" + ALTinterwordstretchfactor * "}" * + write$ newline$ + "\providecommand{\BIBentryALTinterwordspacing}{\spaceskip=\fontdimen2\font plus " + write$ newline$ + "\BIBentryALTinterwordstretchfactor\fontdimen3\font minus \fontdimen4\font\relax}" + write$ newline$ + "\providecommand{\BIBforeignlanguage}[2]{{%" + write$ newline$ + "\expandafter\ifx\csname l@#1\endcsname\relax" + write$ newline$ + "\typeout{** WARNING: IEEEtran.bst: No hyphenation pattern has been}%" + write$ newline$ + "\typeout{** loaded for the language `#1'. Using the pattern for}%" + write$ newline$ + "\typeout{** the default language instead.}%" + write$ newline$ + "\else" + write$ newline$ + "\language=\csname l@#1\endcsname" + write$ newline$ + "\fi" + write$ newline$ + "#2}}" + write$ newline$ + "\providecommand{\BIBdecl}{\relax}" + write$ newline$ + "\BIBdecl" + write$ newline$ +} + +FUNCTION {end.bib} +{ newline$ "\end{thebibliography}" write$ newline$ } + +FUNCTION {if.url.alt.interword.spacing} +{ is.use.alt.interword.spacing + {url empty$ 'skip$ {"\BIBentryALTinterwordspacing" write$ newline$} if$} + { skip$ } + if$ +} + +FUNCTION {if.url.std.interword.spacing} +{ is.use.alt.interword.spacing + {url empty$ 'skip$ {"\BIBentrySTDinterwordspacing" write$ newline$} if$} + { skip$ } + if$ +} + + + + +%%%%%%%%%%%%%%%%%%%%%%%% +%% LONGEST LABEL PASS %% +%%%%%%%%%%%%%%%%%%%%%%%% + +FUNCTION {initialize.longest.label} +{ "" 'longest.label := + #1 'number.label := + #0 'longest.label.width := +} + +FUNCTION {longest.label.pass} +{ type$ "ieeetranbstctl" = + { skip$ } + { number.label int.to.str$ 'label := + number.label #1 + 'number.label := + label width$ longest.label.width > + { label 'longest.label := + label width$ 'longest.label.width := + } + { skip$ } + if$ + } + if$ +} + + + + +%%%%%%%%%%%%%%%%%%%%% +%% FORMAT HANDLERS %% +%%%%%%%%%%%%%%%%%%%%% + +%% Lower Level Formats (used by higher level formats) + +FUNCTION {format.address.org.or.pub.date} +{ 't := + "" + year empty$ + { "empty year in " cite$ * warning$ } + { skip$ } + if$ + address empty$ t empty$ and + year empty$ and month empty$ and + { skip$ } + { this.to.prev.status + this.status.std + cap.status.std + address "address" bibinfo.check * + t empty$ + { skip$ } + { punct.period 'prev.status.punct := + space.large 'prev.status.space := + address empty$ + { skip$ } + { ": " * } + if$ + t * + } + if$ + year empty$ month empty$ and + { skip$ } + { t empty$ address empty$ and + { skip$ } + { ", " * } + if$ + month empty$ + { year empty$ + { skip$ } + { year "year" bibinfo.check * } + if$ + } + { month "month" bibinfo.check * + year empty$ + { skip$ } + { " " * year "year" bibinfo.check * } + if$ + } + if$ + } + if$ + } + if$ +} + + +FUNCTION {format.names} +{ 'bibinfo := + duplicate$ empty$ 'skip$ { + this.to.prev.status + this.status.std + 's := + "" 't := + #1 'nameptr := + s num.names$ 'numnames := + numnames 'namesleft := + { namesleft #0 > } + { s nameptr + name.format.string + format.name$ + bibinfo bibinfo.check + 't := + nameptr #1 > + { nameptr num.names.shown.with.forced.et.al #1 + = + numnames max.num.names.before.forced.et.al > + is.forced.et.al and and + { "others" 't := + #1 'namesleft := + } + { skip$ } + if$ + namesleft #1 > + { ", " * t do.name.latex.cmd * } + { s nameptr "{ll}" format.name$ duplicate$ "others" = + { 't := } + { pop$ } + if$ + t "others" = + { " " * bbl.etal emphasize * } + { numnames #2 > + { "," * } + { skip$ } + if$ + bbl.and + space.word * t do.name.latex.cmd * + } + if$ + } + if$ + } + { t do.name.latex.cmd } + if$ + nameptr #1 + 'nameptr := + namesleft #1 - 'namesleft := + } + while$ + cap.status.std + } if$ +} + + + + +%% Higher Level Formats + +%% addresses/locations + +FUNCTION {format.address} +{ address duplicate$ empty$ 'skip$ + { this.to.prev.status + this.status.std + cap.status.std + } + if$ +} + + + +%% author/editor names + +FUNCTION {format.authors}{ author "author" format.names } + +FUNCTION {format.editors} +{ editor "editor" format.names duplicate$ empty$ 'skip$ + { ", " * + get.bbl.editor + capitalize + * + } + if$ +} + + + +%% date + +FUNCTION {format.date} +{ + month "month" bibinfo.check duplicate$ empty$ + year "year" bibinfo.check duplicate$ empty$ + { swap$ 'skip$ + { this.to.prev.status + this.status.std + cap.status.std + "there's a month but no year in " cite$ * warning$ } + if$ + * + } + { this.to.prev.status + this.status.std + cap.status.std + swap$ 'skip$ + { + swap$ + " " * swap$ + } + if$ + * + } + if$ +} + +FUNCTION {format.date.electronic} +{ month "month" bibinfo.check duplicate$ empty$ + year "year" bibinfo.check duplicate$ empty$ + { swap$ + { pop$ } + { "there's a month but no year in " cite$ * warning$ + pop$ ")" * "(" swap$ * + this.to.prev.status + punct.no 'this.status.punct := + space.normal 'this.status.space := + quote.no 'this.status.quote := + cap.yes 'status.cap := + } + if$ + } + { swap$ + { swap$ pop$ ")" * "(" swap$ * } + { "(" swap$ * ", " * swap$ * ")" * } + if$ + this.to.prev.status + punct.no 'this.status.punct := + space.normal 'this.status.space := + quote.no 'this.status.quote := + cap.yes 'status.cap := + } + if$ +} + + + +%% edition/title + +% Note: IEEE considers the edition to be closely associated with +% the title of a book. So, in IEEEtran.bst the edition is normally handled +% within the formatting of the title. The format.edition function is +% retained here for possible future use. +FUNCTION {format.edition} +{ edition duplicate$ empty$ 'skip$ + { this.to.prev.status + this.status.std + convert.edition + status.cap + { "t" } + { "l" } + if$ change.case$ + "edition" bibinfo.check + "~" * bbl.edition * + cap.status.std + } + if$ +} + +% This is used to format the booktitle of a conference proceedings. +% Here we use the "intype" field to provide the user a way to +% override the word "in" (e.g., with things like "presented at") +% Use of intype stops the emphasis of the booktitle to indicate that +% we no longer mean the written conference proceedings, but the +% conference itself. +FUNCTION {format.in.booktitle} +{ booktitle "booktitle" bibinfo.check duplicate$ empty$ 'skip$ + { this.to.prev.status + this.status.std + select.language + intype missing$ + { emphasize + bbl.in " " * + } + { intype " " * } + if$ + swap$ * + cap.status.std + } + if$ +} + +% This is used to format the booktitle of collection. +% Here the "intype" field is not supported, but "edition" is. +FUNCTION {format.in.booktitle.edition} +{ booktitle "booktitle" bibinfo.check duplicate$ empty$ 'skip$ + { this.to.prev.status + this.status.std + select.language + emphasize + edition empty$ 'skip$ + { ", " * + edition + convert.edition + "l" change.case$ + * "~" * bbl.edition * + } + if$ + bbl.in " " * swap$ * + cap.status.std + } + if$ +} + +FUNCTION {format.article.title} +{ title duplicate$ empty$ 'skip$ + { this.to.prev.status + this.status.std + "t" change.case$ + } + if$ + "title" bibinfo.check + duplicate$ empty$ 'skip$ + { quote.close 'this.status.quote := + is.last.char.not.punct + { punct.std 'this.status.punct := } + { punct.no 'this.status.punct := } + if$ + select.language + "``" swap$ * + cap.status.std + } + if$ +} + +FUNCTION {format.article.title.electronic} +{ title duplicate$ empty$ 'skip$ + { this.to.prev.status + this.status.std + cap.status.std + "t" change.case$ + } + if$ + "title" bibinfo.check + duplicate$ empty$ + { skip$ } + { select.language } + if$ +} + +FUNCTION {format.book.title.edition} +{ title "title" bibinfo.check + duplicate$ empty$ + { "empty title in " cite$ * warning$ } + { this.to.prev.status + this.status.std + select.language + emphasize + edition empty$ 'skip$ + { ", " * + edition + convert.edition + status.cap + { "t" } + { "l" } + if$ + change.case$ + * "~" * bbl.edition * + } + if$ + cap.status.std + } + if$ +} + +FUNCTION {format.book.title} +{ title "title" bibinfo.check + duplicate$ empty$ 'skip$ + { this.to.prev.status + this.status.std + cap.status.std + select.language + emphasize + } + if$ +} + + + +%% journal + +FUNCTION {format.journal} +{ journal duplicate$ empty$ 'skip$ + { this.to.prev.status + this.status.std + cap.status.std + select.language + emphasize + } + if$ +} + + + +%% how published + +FUNCTION {format.howpublished} +{ howpublished duplicate$ empty$ 'skip$ + { this.to.prev.status + this.status.std + cap.status.std + } + if$ +} + + + +%% institutions/organization/publishers/school + +FUNCTION {format.institution} +{ institution duplicate$ empty$ 'skip$ + { this.to.prev.status + this.status.std + cap.status.std + } + if$ +} + +FUNCTION {format.organization} +{ organization duplicate$ empty$ 'skip$ + { this.to.prev.status + this.status.std + cap.status.std + } + if$ +} + +FUNCTION {format.address.publisher.date} +{ publisher "publisher" bibinfo.warn format.address.org.or.pub.date } + +FUNCTION {format.address.publisher.date.nowarn} +{ publisher "publisher" bibinfo.check format.address.org.or.pub.date } + +FUNCTION {format.address.organization.date} +{ organization "organization" bibinfo.check format.address.org.or.pub.date } + +FUNCTION {format.school} +{ school duplicate$ empty$ 'skip$ + { this.to.prev.status + this.status.std + cap.status.std + } + if$ +} + + + +%% volume/number/series/chapter/pages + +FUNCTION {format.volume} +{ volume empty.field.to.null.string + duplicate$ empty$ 'skip$ + { this.to.prev.status + this.status.std + bbl.volume + status.cap + { capitalize } + { skip$ } + if$ + swap$ tie.or.space.prefix + "volume" bibinfo.check + * * + cap.status.std + } + if$ +} + +FUNCTION {format.number} +{ number empty.field.to.null.string + duplicate$ empty$ 'skip$ + { this.to.prev.status + this.status.std + status.cap + { bbl.number capitalize } + { bbl.number } + if$ + swap$ tie.or.space.prefix + "number" bibinfo.check + * * + cap.status.std + } + if$ +} + +FUNCTION {format.number.if.use.for.article} +{ is.use.number.for.article + { format.number } + { "" } + if$ +} + +% IEEE does not seem to tie the series so closely with the volume +% and number as is done in other bibliography styles. Instead the +% series is treated somewhat like an extension of the title. +FUNCTION {format.series} +{ series empty$ + { "" } + { this.to.prev.status + this.status.std + bbl.series " " * + series "series" bibinfo.check * + cap.status.std + } + if$ +} + + +FUNCTION {format.chapter} +{ chapter empty$ + { "" } + { this.to.prev.status + this.status.std + type empty$ + { bbl.chapter } + { type "l" change.case$ + "type" bibinfo.check + } + if$ + chapter tie.or.space.prefix + "chapter" bibinfo.check + * * + cap.status.std + } + if$ +} + + +% The intended use of format.paper is for paper numbers of inproceedings. +% The paper type can be overridden via the type field. +% We allow the type to be displayed even if the paper number is absent +% for things like "postdeadline paper" +FUNCTION {format.paper} +{ is.use.paper + { paper empty$ + { type empty$ + { "" } + { this.to.prev.status + this.status.std + type "type" bibinfo.check + cap.status.std + } + if$ + } + { this.to.prev.status + this.status.std + type empty$ + { bbl.paper } + { type "type" bibinfo.check } + if$ + " " * paper + "paper" bibinfo.check + * + cap.status.std + } + if$ + } + { "" } + if$ +} + + +FUNCTION {format.pages} +{ pages duplicate$ empty$ 'skip$ + { this.to.prev.status + this.status.std + duplicate$ is.multiple.pages + { + bbl.pages swap$ + n.dashify + } + { + bbl.page swap$ + } + if$ + tie.or.space.prefix + "pages" bibinfo.check + * * + cap.status.std + } + if$ +} + + + +%% technical report number + +FUNCTION {format.tech.report.number} +{ number "number" bibinfo.check + this.to.prev.status + this.status.std + cap.status.std + type duplicate$ empty$ + { pop$ + bbl.techrep + } + { skip$ } + if$ + "type" bibinfo.check + swap$ duplicate$ empty$ + { pop$ } + { tie.or.space.prefix * * } + if$ +} + + + +%% note + +FUNCTION {format.note} +{ note empty$ + { "" } + { this.to.prev.status + this.status.std + punct.period 'this.status.punct := + note #1 #1 substring$ + duplicate$ "{" = + { skip$ } + { status.cap + { "u" } + { "l" } + if$ + change.case$ + } + if$ + note #2 global.max$ substring$ * "note" bibinfo.check + cap.yes 'status.cap := + } + if$ +} + + + +%% patent + +FUNCTION {format.patent.date} +{ this.to.prev.status + this.status.std + year empty$ + { monthfiled duplicate$ empty$ + { "monthfiled" bibinfo.check pop$ "" } + { "monthfiled" bibinfo.check } + if$ + dayfiled duplicate$ empty$ + { "dayfiled" bibinfo.check pop$ "" * } + { "dayfiled" bibinfo.check + monthfiled empty$ + { "dayfiled without a monthfiled in " cite$ * warning$ + * + } + { " " swap$ * * } + if$ + } + if$ + yearfiled empty$ + { "no year or yearfiled in " cite$ * warning$ } + { yearfiled "yearfiled" bibinfo.check + swap$ + duplicate$ empty$ + { pop$ } + { ", " * swap$ * } + if$ + } + if$ + } + { month duplicate$ empty$ + { "month" bibinfo.check pop$ "" } + { "month" bibinfo.check } + if$ + day duplicate$ empty$ + { "day" bibinfo.check pop$ "" * } + { "day" bibinfo.check + month empty$ + { "day without a month in " cite$ * warning$ + * + } + { " " swap$ * * } + if$ + } + if$ + year "year" bibinfo.check + swap$ + duplicate$ empty$ + { pop$ } + { ", " * swap$ * } + if$ + } + if$ + cap.status.std +} + +FUNCTION {format.patent.nationality.type.number} +{ this.to.prev.status + this.status.std + nationality duplicate$ empty$ + { "nationality" bibinfo.warn pop$ "" } + { "nationality" bibinfo.check + duplicate$ "l" change.case$ "united states" = + { pop$ bbl.patentUS } + { skip$ } + if$ + " " * + } + if$ + type empty$ + { bbl.patent "type" bibinfo.check } + { type "type" bibinfo.check } + if$ + * + number duplicate$ empty$ + { "number" bibinfo.warn pop$ } + { "number" bibinfo.check + large.number.separate + swap$ " " * swap$ * + } + if$ + cap.status.std +} + + + +%% standard + +FUNCTION {format.organization.institution.standard.type.number} +{ this.to.prev.status + this.status.std + organization duplicate$ empty$ + { pop$ + institution duplicate$ empty$ + { "institution" bibinfo.warn } + { "institution" bibinfo.warn " " * } + if$ + } + { "organization" bibinfo.warn " " * } + if$ + type empty$ + { bbl.standard "type" bibinfo.check } + { type "type" bibinfo.check } + if$ + * + number duplicate$ empty$ + { "number" bibinfo.check pop$ } + { "number" bibinfo.check + large.number.separate + swap$ " " * swap$ * + } + if$ + cap.status.std +} + +FUNCTION {format.revision} +{ revision empty$ + { "" } + { this.to.prev.status + this.status.std + bbl.revision + revision tie.or.space.prefix + "revision" bibinfo.check + * * + cap.status.std + } + if$ +} + + +%% thesis + +FUNCTION {format.master.thesis.type} +{ this.to.prev.status + this.status.std + type empty$ + { + bbl.mthesis + } + { + type "type" bibinfo.check + } + if$ +cap.status.std +} + +FUNCTION {format.phd.thesis.type} +{ this.to.prev.status + this.status.std + type empty$ + { + bbl.phdthesis + } + { + type "type" bibinfo.check + } + if$ +cap.status.std +} + + + +%% URL + +FUNCTION {format.url} +{ url empty$ + { "" } + { this.to.prev.status + this.status.std + cap.yes 'status.cap := + name.url.prefix " " * + "\url{" * url * "}" * + punct.no 'this.status.punct := + punct.period 'prev.status.punct := + space.normal 'this.status.space := + space.normal 'prev.status.space := + quote.no 'this.status.quote := + } + if$ +} + + + + +%%%%%%%%%%%%%%%%%%%% +%% ENTRY HANDLERS %% +%%%%%%%%%%%%%%%%%%%% + + +% Note: In many journals, IEEE (or the authors) tend not to show the number +% for articles, so the display of the number is controlled here by the +% switch "is.use.number.for.article" +FUNCTION {article} +{ std.status.using.comma + start.entry + if.url.alt.interword.spacing + format.authors "author" output.warn + name.or.dash + format.article.title "title" output.warn + format.journal "journal" bibinfo.check "journal" output.warn + format.volume output + format.number.if.use.for.article output + format.pages output + format.date "year" output.warn + format.note output + format.url output + fin.entry + if.url.std.interword.spacing +} + +FUNCTION {book} +{ std.status.using.comma + start.entry + if.url.alt.interword.spacing + author empty$ + { format.editors "author and editor" output.warn } + { format.authors output.nonnull } + if$ + name.or.dash + format.book.title.edition output + format.series output + author empty$ + { skip$ } + { format.editors output } + if$ + format.address.publisher.date output + format.volume output + format.number output + format.note output + format.url output + fin.entry + if.url.std.interword.spacing +} + +FUNCTION {booklet} +{ std.status.using.comma + start.entry + if.url.alt.interword.spacing + format.authors output + name.or.dash + format.article.title "title" output.warn + format.howpublished "howpublished" bibinfo.check output + format.organization "organization" bibinfo.check output + format.address "address" bibinfo.check output + format.date output + format.note output + format.url output + fin.entry + if.url.std.interword.spacing +} + +FUNCTION {electronic} +{ std.status.using.period + start.entry + if.url.alt.interword.spacing + format.authors output + name.or.dash + format.date.electronic output + format.article.title.electronic output + format.howpublished "howpublished" bibinfo.check output + format.organization "organization" bibinfo.check output + format.address "address" bibinfo.check output + format.note output + format.url output + fin.entry + empty.entry.warn + if.url.std.interword.spacing +} + +FUNCTION {inbook} +{ std.status.using.comma + start.entry + if.url.alt.interword.spacing + author empty$ + { format.editors "author and editor" output.warn } + { format.authors output.nonnull } + if$ + name.or.dash + format.book.title.edition output + format.series output + format.address.publisher.date output + format.volume output + format.number output + format.chapter output + format.pages output + format.note output + format.url output + fin.entry + if.url.std.interword.spacing +} + +FUNCTION {incollection} +{ std.status.using.comma + start.entry + if.url.alt.interword.spacing + format.authors "author" output.warn + name.or.dash + format.article.title "title" output.warn + format.in.booktitle.edition "booktitle" output.warn + format.series output + format.editors output + format.address.publisher.date.nowarn output + format.volume output + format.number output + format.chapter output + format.pages output + format.note output + format.url output + fin.entry + if.url.std.interword.spacing +} + +FUNCTION {inproceedings} +{ std.status.using.comma + start.entry + if.url.alt.interword.spacing + format.authors "author" output.warn + name.or.dash + format.article.title "title" output.warn + format.in.booktitle "booktitle" output.warn + format.series output + format.editors output + format.volume output + format.number output + publisher empty$ + { format.address.organization.date output } + { format.organization "organization" bibinfo.check output + format.address.publisher.date output + } + if$ + format.paper output + format.pages output + format.note output + format.url output + fin.entry + if.url.std.interword.spacing +} + +FUNCTION {manual} +{ std.status.using.comma + start.entry + if.url.alt.interword.spacing + format.authors output + name.or.dash + format.book.title.edition "title" output.warn + format.howpublished "howpublished" bibinfo.check output + format.organization "organization" bibinfo.check output + format.address "address" bibinfo.check output + format.date output + format.note output + format.url output + fin.entry + if.url.std.interword.spacing +} + +FUNCTION {mastersthesis} +{ std.status.using.comma + start.entry + if.url.alt.interword.spacing + format.authors "author" output.warn + name.or.dash + format.article.title "title" output.warn + format.master.thesis.type output.nonnull + format.school "school" bibinfo.warn output + format.address "address" bibinfo.check output + format.date "year" output.warn + format.note output + format.url output + fin.entry + if.url.std.interword.spacing +} + +FUNCTION {misc} +{ std.status.using.comma + start.entry + if.url.alt.interword.spacing + format.authors output + name.or.dash + format.article.title output + format.howpublished "howpublished" bibinfo.check output + format.organization "organization" bibinfo.check output + format.address "address" bibinfo.check output + format.pages output + format.date output + format.note output + format.url output + fin.entry + empty.entry.warn + if.url.std.interword.spacing +} + +FUNCTION {patent} +{ std.status.using.comma + start.entry + if.url.alt.interword.spacing + format.authors output + name.or.dash + format.article.title output + format.patent.nationality.type.number output + format.patent.date output + format.note output + format.url output + fin.entry + empty.entry.warn + if.url.std.interword.spacing +} + +FUNCTION {periodical} +{ std.status.using.comma + start.entry + if.url.alt.interword.spacing + format.editors output + name.or.dash + format.book.title "title" output.warn + format.series output + format.volume output + format.number output + format.organization "organization" bibinfo.check output + format.date "year" output.warn + format.note output + format.url output + fin.entry + if.url.std.interword.spacing +} + +FUNCTION {phdthesis} +{ std.status.using.comma + start.entry + if.url.alt.interword.spacing + format.authors "author" output.warn + name.or.dash + format.article.title "title" output.warn + format.phd.thesis.type output.nonnull + format.school "school" bibinfo.warn output + format.address "address" bibinfo.check output + format.date "year" output.warn + format.note output + format.url output + fin.entry + if.url.std.interword.spacing +} + +FUNCTION {proceedings} +{ std.status.using.comma + start.entry + if.url.alt.interword.spacing + format.editors output + name.or.dash + format.book.title "title" output.warn + format.series output + format.volume output + format.number output + publisher empty$ + { format.address.organization.date output } + { format.organization "organization" bibinfo.check output + format.address.publisher.date output + } + if$ + format.note output + format.url output + fin.entry + if.url.std.interword.spacing +} + +FUNCTION {standard} +{ std.status.using.comma + start.entry + if.url.alt.interword.spacing + format.authors output + name.or.dash + format.book.title "title" output.warn + format.howpublished "howpublished" bibinfo.check output + format.organization.institution.standard.type.number output + format.revision output + format.date output + format.note output + format.url output + fin.entry + if.url.std.interword.spacing +} + +FUNCTION {techreport} +{ std.status.using.comma + start.entry + if.url.alt.interword.spacing + format.authors "author" output.warn + name.or.dash + format.article.title "title" output.warn + format.howpublished "howpublished" bibinfo.check output + format.institution "institution" bibinfo.warn output + format.address "address" bibinfo.check output + format.tech.report.number output.nonnull + format.date "year" output.warn + format.note output + format.url output + fin.entry + if.url.std.interword.spacing +} + +FUNCTION {unpublished} +{ std.status.using.comma + start.entry + if.url.alt.interword.spacing + format.authors "author" output.warn + name.or.dash + format.article.title "title" output.warn + format.date output + format.note "note" output.warn + format.url output + fin.entry + if.url.std.interword.spacing +} + + +% The special entry type which provides the user interface to the +% BST controls +FUNCTION {IEEEtranBSTCTL} +{ is.print.banners.to.terminal + { "** IEEEtran BST control entry " quote$ * cite$ * quote$ * " detected." * + top$ + } + { skip$ } + if$ + CTLuse_article_number + empty$ + { skip$ } + { CTLuse_article_number + yes.no.to.int + 'is.use.number.for.article := + } + if$ + CTLuse_paper + empty$ + { skip$ } + { CTLuse_paper + yes.no.to.int + 'is.use.paper := + } + if$ + CTLuse_forced_etal + empty$ + { skip$ } + { CTLuse_forced_etal + yes.no.to.int + 'is.forced.et.al := + } + if$ + CTLmax_names_forced_etal + empty$ + { skip$ } + { CTLmax_names_forced_etal + string.to.integer + 'max.num.names.before.forced.et.al := + } + if$ + CTLnames_show_etal + empty$ + { skip$ } + { CTLnames_show_etal + string.to.integer + 'num.names.shown.with.forced.et.al := + } + if$ + CTLuse_alt_spacing + empty$ + { skip$ } + { CTLuse_alt_spacing + yes.no.to.int + 'is.use.alt.interword.spacing := + } + if$ + CTLalt_stretch_factor + empty$ + { skip$ } + { CTLalt_stretch_factor + 'ALTinterwordstretchfactor := + "\renewcommand{\BIBentryALTinterwordstretchfactor}{" + ALTinterwordstretchfactor * "}" * + write$ newline$ + } + if$ + CTLdash_repeated_names + empty$ + { skip$ } + { CTLdash_repeated_names + yes.no.to.int + 'is.dash.repeated.names := + } + if$ + CTLname_format_string + empty$ + { skip$ } + { CTLname_format_string + 'name.format.string := + } + if$ + CTLname_latex_cmd + empty$ + { skip$ } + { CTLname_latex_cmd + 'name.latex.cmd := + } + if$ + CTLname_url_prefix + missing$ + { skip$ } + { CTLname_url_prefix + 'name.url.prefix := + } + if$ + + + num.names.shown.with.forced.et.al max.num.names.before.forced.et.al > + { "CTLnames_show_etal cannot be greater than CTLmax_names_forced_etal in " cite$ * warning$ + max.num.names.before.forced.et.al 'num.names.shown.with.forced.et.al := + } + { skip$ } + if$ +} + + +%%%%%%%%%%%%%%%%%%% +%% ENTRY ALIASES %% +%%%%%%%%%%%%%%%%%%% +FUNCTION {conference}{inproceedings} +FUNCTION {online}{electronic} +FUNCTION {internet}{electronic} +FUNCTION {webpage}{electronic} +FUNCTION {www}{electronic} +FUNCTION {default.type}{misc} + + + +%%%%%%%%%%%%%%%%%% +%% MAIN PROGRAM %% +%%%%%%%%%%%%%%%%%% + +READ + +EXECUTE {initialize.controls} +EXECUTE {initialize.status.constants} +EXECUTE {banner.message} + +EXECUTE {initialize.longest.label} +ITERATE {longest.label.pass} + +EXECUTE {begin.bib} +ITERATE {call.type$} +EXECUTE {end.bib} + +EXECUTE{completed.message} + + +%% That's all folks, mds. diff --git a/cometbft/v0.38/spec/consensus/consensus-paper/IEEEtran.cls b/cometbft/v0.38/spec/consensus/consensus-paper/IEEEtran.cls new file mode 100644 index 00000000..9c967d55 --- /dev/null +++ b/cometbft/v0.38/spec/consensus/consensus-paper/IEEEtran.cls @@ -0,0 +1,4733 @@ +%% +%% IEEEtran.cls 2011/11/03 version V1.8 based on +%% IEEEtran.cls 2007/03/05 version V1.7a +%% The changes in V1.8 are made with a single goal in mind: +%% to change the look of the output using the [conference] option +%% and the default font size (10pt) to match the Word template more closely. +%% These changes may well have undesired side effects when other options +%% are in force! +%% +%% +%% This is the official IEEE LaTeX class for authors of the Institute of +%% Electrical and Electronics Engineers (IEEE) Transactions journals and +%% conferences. +%% +%% Support sites: +%% http://www.michaelshell.org/tex/ieeetran/ +%% http://www.ctan.org/tex-archive/macros/latex/contrib/IEEEtran/ +%% and +%% http://www.ieee.org/ +%% +%% Based on the original 1993 IEEEtran.cls, but with many bug fixes +%% and enhancements (from both JVH and MDS) over the 1996/7 version. +%% +%% +%% Contributors: +%% Gerry Murray (1993), Silvano Balemi (1993), +%% Jon Dixon (1996), Peter N"uchter (1996), +%% Juergen von Hagen (2000), and Michael Shell (2001-2007) +%% +%% +%% Copyright (c) 1993-2000 by Gerry Murray, Silvano Balemi, +%% Jon Dixon, Peter N"uchter, +%% Juergen von Hagen +%% and +%% Copyright (c) 2001-2007 by Michael Shell +%% +%% Current maintainer (V1.3 to V1.7): Michael Shell +%% See: +%% http://www.michaelshell.org/ +%% for current contact information. +%% +%% Special thanks to Peter Wilson (CUA) and Donald Arseneau +%% for allowing the inclusion of the \@ifmtarg command +%% from their ifmtarg LaTeX package. +%% +%%************************************************************************* +%% Legal Notice: +%% This code is offered as-is without any warranty either expressed or +%% implied; without even the implied warranty of MERCHANTABILITY or +%% FITNESS FOR A PARTICULAR PURPOSE! +%% User assumes all risk. +%% In no event shall IEEE or any contributor to this code be liable for +%% any damages or losses, including, but not limited to, incidental, +%% consequential, or any other damages, resulting from the use or misuse +%% of any information contained here. +%% +%% All comments are the opinions of their respective authors and are not +%% necessarily endorsed by the IEEE. +%% +%% This work is distributed under the LaTeX Project Public License (LPPL) +%% ( http://www.latex-project.org/ ) version 1.3, and may be freely used, +%% distributed and modified. A copy of the LPPL, version 1.3, is included +%% in the base LaTeX documentation of all distributions of LaTeX released +%% 2003/12/01 or later. +%% Retain all contribution notices and credits. +%% ** Modified files should be clearly indicated as such, including ** +%% ** renaming them and changing author support contact information. ** +%% +%% File list of work: IEEEtran.cls, IEEEtran_HOWTO.pdf, bare_adv.tex, +%% bare_conf.tex, bare_jrnl.tex, bare_jrnl_compsoc.tex +%% +%% Major changes to the user interface should be indicated by an +%% increase in the version numbers. If a version is a beta, it will +%% be indicated with a BETA suffix, i.e., 1.4 BETA. +%% Small changes can be indicated by appending letters to the version +%% such as "IEEEtran_v14a.cls". +%% In all cases, \Providesclass, any \typeout messages to the user, +%% \IEEEtransversionmajor and \IEEEtransversionminor must reflect the +%% correct version information. +%% The changes should also be documented via source comments. +%%************************************************************************* +%% +% +% Available class options +% e.g., \documentclass[10pt,conference]{IEEEtran} +% +% *** choose only one from each category *** +% +% 9pt, 10pt, 11pt, 12pt +% Sets normal font size. The default is 10pt. +% +% conference, journal, technote, peerreview, peerreviewca +% determines format mode - conference papers, journal papers, +% correspondence papers (technotes), or peer review papers. The user +% should also select 9pt when using technote. peerreview is like +% journal mode, but provides for a single-column "cover" title page for +% anonymous peer review. The paper title (without the author names) is +% repeated at the top of the page after the cover page. For peer review +% papers, the \IEEEpeerreviewmaketitle command must be executed (will +% automatically be ignored for non-peerreview modes) at the place the +% cover page is to end, usually just after the abstract (keywords are +% not normally used with peer review papers). peerreviewca is like +% peerreview, but allows the author names to be entered and formatted +% as with conference mode so that author affiliation and contact +% information can be easily seen on the cover page. +% The default is journal. +% +% draft, draftcls, draftclsnofoot, final +% determines if paper is formatted as a widely spaced draft (for +% handwritten editor comments) or as a properly typeset final version. +% draftcls restricts draft mode to the class file while all other LaTeX +% packages (i.e., \usepackage{graphicx}) will behave as final - allows +% for a draft paper with visible figures, etc. draftclsnofoot is like +% draftcls, but does not display the date and the word "DRAFT" at the foot +% of the pages. If using one of the draft modes, the user will probably +% also want to select onecolumn. +% The default is final. +% +% letterpaper, a4paper +% determines paper size: 8.5in X 11in or 210mm X 297mm. CHANGING THE PAPER +% SIZE WILL NOT ALTER THE TYPESETTING OF THE DOCUMENT - ONLY THE MARGINS +% WILL BE AFFECTED. In particular, documents using the a4paper option will +% have reduced side margins (A4 is narrower than US letter) and a longer +% bottom margin (A4 is longer than US letter). For both cases, the top +% margins will be the same and the text will be horizontally centered. +% For final submission to IEEE, authors should use US letter (8.5 X 11in) +% paper. Note that authors should ensure that all post-processing +% (ps, pdf, etc.) uses the same paper specificiation as the .tex document. +% Problems here are by far the number one reason for incorrect margins. +% IEEEtran will automatically set the default paper size under pdflatex +% (without requiring a change to pdftex.cfg), so this issue is more +% important to dvips users. Fix config.ps, config.pdf, or ~/.dvipsrc for +% dvips, or use the dvips -t papersize option instead as needed. See the +% testflow documentation +% http://www.ctan.org/tex-archive/macros/latex/contrib/IEEEtran/testflow +% for more details on dvips paper size configuration. +% The default is letterpaper. +% +% oneside, twoside +% determines if layout follows single sided or two sided (duplex) +% printing. The only notable change is with the headings at the top of +% the pages. +% The default is oneside. +% +% onecolumn, twocolumn +% determines if text is organized into one or two columns per page. One +% column mode is usually used only with draft papers. +% The default is twocolumn. +% +% compsoc +% Use the format of the IEEE Computer Society. +% +% romanappendices +% Use the "Appendix I" convention when numbering appendices. IEEEtran.cls +% now defaults to Alpha "Appendix A" convention - the opposite of what +% v1.6b and earlier did. +% +% captionsoff +% disables the display of the figure/table captions. Some IEEE journals +% request that captions be removed and figures/tables be put on pages +% of their own at the end of an initial paper submission. The endfloat +% package can be used with this class option to achieve this format. +% +% nofonttune +% turns off tuning of the font interword spacing. Maybe useful to those +% not using the standard Times fonts or for those who have already "tuned" +% their fonts. +% The default is to enable IEEEtran to tune font parameters. +% +% +%---------- +% Available CLASSINPUTs provided (all are macros unless otherwise noted): +% \CLASSINPUTbaselinestretch +% \CLASSINPUTinnersidemargin +% \CLASSINPUToutersidemargin +% \CLASSINPUTtoptextmargin +% \CLASSINPUTbottomtextmargin +% +% Available CLASSINFOs provided: +% \ifCLASSINFOpdf (TeX if conditional) +% \CLASSINFOpaperwidth (macro) +% \CLASSINFOpaperheight (macro) +% \CLASSINFOnormalsizebaselineskip (length) +% \CLASSINFOnormalsizeunitybaselineskip (length) +% +% Available CLASSOPTIONs provided: +% all class option flags (TeX if conditionals) unless otherwise noted, +% e.g., \ifCLASSOPTIONcaptionsoff +% point size options provided as a single macro: +% \CLASSOPTIONpt +% which will be defined as 9, 10, 11, or 12 depending on the document's +% normalsize point size. +% also, class option peerreviewca implies the use of class option peerreview +% and classoption draft implies the use of class option draftcls + + + + + +\ProvidesClass{IEEEtran}[2012/11/21 V1.8c by Harald Hanche-Olsen and Anders Christensen] +\typeout{-- Based on V1.7a by Michael Shell} +\typeout{-- See the "IEEEtran_HOWTO" manual for usage information.} +\typeout{-- http://www.michaelshell.org/tex/ieeetran/} +\NeedsTeXFormat{LaTeX2e} + +% IEEEtran.cls version numbers, provided as of V1.3 +% These values serve as a way a .tex file can +% determine if the new features are provided. +% The version number of this IEEEtrans.cls can be obtained from +% these values. i.e., V1.4 +% KEEP THESE AS INTEGERS! i.e., NO {4a} or anything like that- +% (no need to enumerate "a" minor changes here) +\def\IEEEtransversionmajor{1} +\def\IEEEtransversionminor{7} + +% These do nothing, but provide them like in article.cls +\newif\if@restonecol +\newif\if@titlepage + + +% class option conditionals +\newif\ifCLASSOPTIONonecolumn \CLASSOPTIONonecolumnfalse +\newif\ifCLASSOPTIONtwocolumn \CLASSOPTIONtwocolumntrue + +\newif\ifCLASSOPTIONoneside \CLASSOPTIONonesidetrue +\newif\ifCLASSOPTIONtwoside \CLASSOPTIONtwosidefalse + +\newif\ifCLASSOPTIONfinal \CLASSOPTIONfinaltrue +\newif\ifCLASSOPTIONdraft \CLASSOPTIONdraftfalse +\newif\ifCLASSOPTIONdraftcls \CLASSOPTIONdraftclsfalse +\newif\ifCLASSOPTIONdraftclsnofoot \CLASSOPTIONdraftclsnofootfalse + +\newif\ifCLASSOPTIONpeerreview \CLASSOPTIONpeerreviewfalse +\newif\ifCLASSOPTIONpeerreviewca \CLASSOPTIONpeerreviewcafalse + +\newif\ifCLASSOPTIONjournal \CLASSOPTIONjournaltrue +\newif\ifCLASSOPTIONconference \CLASSOPTIONconferencefalse +\newif\ifCLASSOPTIONtechnote \CLASSOPTIONtechnotefalse + +\newif\ifCLASSOPTIONnofonttune \CLASSOPTIONnofonttunefalse + +\newif\ifCLASSOPTIONcaptionsoff \CLASSOPTIONcaptionsofffalse + +\newif\ifCLASSOPTIONcompsoc \CLASSOPTIONcompsocfalse + +\newif\ifCLASSOPTIONromanappendices \CLASSOPTIONromanappendicesfalse + + +% class info conditionals + +% indicates if pdf (via pdflatex) output +\newif\ifCLASSINFOpdf \CLASSINFOpdffalse + + +% V1.6b internal flag to show if using a4paper +\newif\if@IEEEusingAfourpaper \@IEEEusingAfourpaperfalse + + + +% IEEEtran class scratch pad registers +% dimen +\newdimen\@IEEEtrantmpdimenA +\newdimen\@IEEEtrantmpdimenB +% count +\newcount\@IEEEtrantmpcountA +\newcount\@IEEEtrantmpcountB +% token list +\newtoks\@IEEEtrantmptoksA + +% we use \CLASSOPTIONpt so that we can ID the point size (even for 9pt docs) +% as well as LaTeX's \@ptsize to retain some compatability with some +% external packages +\def\@ptsize{0} +% LaTeX does not support 9pt, so we set \@ptsize to 0 - same as that of 10pt +\DeclareOption{9pt}{\def\CLASSOPTIONpt{9}\def\@ptsize{0}} +\DeclareOption{10pt}{\def\CLASSOPTIONpt{10}\def\@ptsize{0}} +\DeclareOption{11pt}{\def\CLASSOPTIONpt{11}\def\@ptsize{1}} +\DeclareOption{12pt}{\def\CLASSOPTIONpt{12}\def\@ptsize{2}} + + + +\DeclareOption{letterpaper}{\setlength{\paperheight}{11in}% + \setlength{\paperwidth}{8.5in}% + \@IEEEusingAfourpaperfalse + \def\CLASSOPTIONpaper{letter}% + \def\CLASSINFOpaperwidth{8.5in}% + \def\CLASSINFOpaperheight{11in}} + + +\DeclareOption{a4paper}{\setlength{\paperheight}{297mm}% + \setlength{\paperwidth}{210mm}% + \@IEEEusingAfourpapertrue + \def\CLASSOPTIONpaper{a4}% + \def\CLASSINFOpaperwidth{210mm}% + \def\CLASSINFOpaperheight{297mm}} + +\DeclareOption{oneside}{\@twosidefalse\@mparswitchfalse + \CLASSOPTIONonesidetrue\CLASSOPTIONtwosidefalse} +\DeclareOption{twoside}{\@twosidetrue\@mparswitchtrue + \CLASSOPTIONtwosidetrue\CLASSOPTIONonesidefalse} + +\DeclareOption{onecolumn}{\CLASSOPTIONonecolumntrue\CLASSOPTIONtwocolumnfalse} +\DeclareOption{twocolumn}{\CLASSOPTIONtwocolumntrue\CLASSOPTIONonecolumnfalse} + +% If the user selects draft, then this class AND any packages +% will go into draft mode. +\DeclareOption{draft}{\CLASSOPTIONdrafttrue\CLASSOPTIONdraftclstrue + \CLASSOPTIONdraftclsnofootfalse} +% draftcls is for a draft mode which will not affect any packages +% used by the document. +\DeclareOption{draftcls}{\CLASSOPTIONdraftfalse\CLASSOPTIONdraftclstrue + \CLASSOPTIONdraftclsnofootfalse} +% draftclsnofoot is like draftcls, but without the footer. +\DeclareOption{draftclsnofoot}{\CLASSOPTIONdraftfalse\CLASSOPTIONdraftclstrue + \CLASSOPTIONdraftclsnofoottrue} +\DeclareOption{final}{\CLASSOPTIONdraftfalse\CLASSOPTIONdraftclsfalse + \CLASSOPTIONdraftclsnofootfalse} + +\DeclareOption{journal}{\CLASSOPTIONpeerreviewfalse\CLASSOPTIONpeerreviewcafalse + \CLASSOPTIONjournaltrue\CLASSOPTIONconferencefalse\CLASSOPTIONtechnotefalse} + +\DeclareOption{conference}{\CLASSOPTIONpeerreviewfalse\CLASSOPTIONpeerreviewcafalse + \CLASSOPTIONjournalfalse\CLASSOPTIONconferencetrue\CLASSOPTIONtechnotefalse} + +\DeclareOption{technote}{\CLASSOPTIONpeerreviewfalse\CLASSOPTIONpeerreviewcafalse + \CLASSOPTIONjournalfalse\CLASSOPTIONconferencefalse\CLASSOPTIONtechnotetrue} + +\DeclareOption{peerreview}{\CLASSOPTIONpeerreviewtrue\CLASSOPTIONpeerreviewcafalse + \CLASSOPTIONjournalfalse\CLASSOPTIONconferencefalse\CLASSOPTIONtechnotefalse} + +\DeclareOption{peerreviewca}{\CLASSOPTIONpeerreviewtrue\CLASSOPTIONpeerreviewcatrue + \CLASSOPTIONjournalfalse\CLASSOPTIONconferencefalse\CLASSOPTIONtechnotefalse} + +\DeclareOption{nofonttune}{\CLASSOPTIONnofonttunetrue} + +\DeclareOption{captionsoff}{\CLASSOPTIONcaptionsofftrue} + +\DeclareOption{compsoc}{\CLASSOPTIONcompsoctrue} + +\DeclareOption{romanappendices}{\CLASSOPTIONromanappendicestrue} + + +% default to US letter paper, 10pt, twocolumn, one sided, final, journal +\ExecuteOptions{letterpaper,10pt,twocolumn,oneside,final,journal} +% overrride these defaults per user requests +\ProcessOptions + + + +% Computer Society conditional execution command +\long\def\@IEEEcompsoconly#1{\relax\ifCLASSOPTIONcompsoc\relax#1\relax\fi\relax} +% inverse +\long\def\@IEEEnotcompsoconly#1{\relax\ifCLASSOPTIONcompsoc\else\relax#1\relax\fi\relax} +% compsoc conference +\long\def\@IEEEcompsocconfonly#1{\relax\ifCLASSOPTIONcompsoc\ifCLASSOPTIONconference\relax#1\relax\fi\fi\relax} +% compsoc not conference +\long\def\@IEEEcompsocnotconfonly#1{\relax\ifCLASSOPTIONcompsoc\ifCLASSOPTIONconference\else\relax#1\relax\fi\fi\relax} + + +% IEEE uses Times Roman font, so we'll default to Times. +% These three commands make up the entire times.sty package. +\renewcommand{\sfdefault}{phv} +\renewcommand{\rmdefault}{ptm} +\renewcommand{\ttdefault}{pcr} + +\@IEEEcompsoconly{\typeout{-- Using IEEE Computer Society mode.}} + +% V1.7 compsoc nonconference papers, use Palatino/Palladio as the main text font, +% not Times Roman. +\@IEEEcompsocnotconfonly{\renewcommand{\rmdefault}{ppl}} + +% enable Times/Palatino main text font +\normalfont\selectfont + + + + + +% V1.7 conference notice message hook +\def\@IEEEconsolenoticeconference{\typeout{}% +\typeout{** Conference Paper **}% +\typeout{Before submitting the final camera ready copy, remember to:}% +\typeout{}% +\typeout{ 1. Manually equalize the lengths of two columns on the last page}% +\typeout{ of your paper;}% +\typeout{}% +\typeout{ 2. Ensure that any PostScript and/or PDF output post-processing}% +\typeout{ uses only Type 1 fonts and that every step in the generation}% +\typeout{ process uses the appropriate paper size.}% +\typeout{}} + + +% we can send console reminder messages to the user here +\AtEndDocument{\ifCLASSOPTIONconference\@IEEEconsolenoticeconference\fi} + + +% warn about the use of single column other than for draft mode +\ifCLASSOPTIONtwocolumn\else% + \ifCLASSOPTIONdraftcls\else% + \typeout{** ATTENTION: Single column mode is not typically used with IEEE publications.}% + \fi% +\fi + + +% V1.7 improved paper size setting code. +% Set pdfpage and dvips paper sizes. Conditional tests are similar to that +% of ifpdf.sty. Retain within {} to ensure tested macros are never altered, +% even if only effect is to set them to \relax. +% if \pdfoutput is undefined or equal to relax, output a dvips special +{\@ifundefined{pdfoutput}{\AtBeginDvi{\special{papersize=\CLASSINFOpaperwidth,\CLASSINFOpaperheight}}}{% +% pdfoutput is defined and not equal to \relax +% check for pdfpageheight existence just in case someone sets pdfoutput +% under non-pdflatex. If exists, set them regardless of value of \pdfoutput. +\@ifundefined{pdfpageheight}{\relax}{\global\pdfpagewidth\paperwidth +\global\pdfpageheight\paperheight}% +% if using \pdfoutput=0 under pdflatex, send dvips papersize special +\ifcase\pdfoutput +\AtBeginDvi{\special{papersize=\CLASSINFOpaperwidth,\CLASSINFOpaperheight}}% +\else +% we are using pdf output, set CLASSINFOpdf flag +\global\CLASSINFOpdftrue +\fi}} + +% let the user know the selected papersize +\typeout{-- Using \CLASSINFOpaperwidth\space x \CLASSINFOpaperheight\space +(\CLASSOPTIONpaper)\space paper.} + +\ifCLASSINFOpdf +\typeout{-- Using PDF output.} +\else +\typeout{-- Using DVI output.} +\fi + + +% The idea hinted here is for LaTeX to generate markleft{} and markright{} +% automatically for you after you enter \author{}, \journal{}, +% \journaldate{}, journalvol{}, \journalnum{}, etc. +% However, there may be some backward compatibility issues here as +% well as some special applications for IEEEtran.cls and special issues +% that may require the flexible \markleft{}, \markright{} and/or \markboth{}. +% We'll leave this as an open future suggestion. +%\newcommand{\journal}[1]{\def\@journal{#1}} +%\def\@journal{} + + + +% pointsize values +% used with ifx to determine the document's normal size +\def\@IEEEptsizenine{9} +\def\@IEEEptsizeten{10} +\def\@IEEEptsizeeleven{11} +\def\@IEEEptsizetwelve{12} + + + +% FONT DEFINITIONS (No sizexx.clo file needed) +% V1.6 revised font sizes, displayskip values and +% revised normalsize baselineskip to reduce underfull vbox problems +% on the 58pc = 696pt = 9.5in text height we want +% normalsize #lines/column baselineskip (aka leading) +% 9pt 63 11.0476pt (truncated down) +% 10pt 58 12pt (exact) +% 11pt 52 13.3846pt (truncated down) +% 12pt 50 13.92pt (exact) +% + +% we need to store the nominal baselineskip for the given font size +% in case baselinestretch ever changes. +% this is a dimen, so it will not hold stretch or shrink +\newdimen\@IEEEnormalsizeunitybaselineskip +\@IEEEnormalsizeunitybaselineskip\baselineskip + +\ifx\CLASSOPTIONpt\@IEEEptsizenine +\typeout{-- This is a 9 point document.} +\def\normalsize{\@setfontsize{\normalsize}{9}{11.0476pt}}% +\setlength{\@IEEEnormalsizeunitybaselineskip}{11.0476pt}% +\normalsize +\abovedisplayskip 1.5ex plus3pt minus1pt% +\belowdisplayskip \abovedisplayskip% +\abovedisplayshortskip 0pt plus3pt% +\belowdisplayshortskip 1.5ex plus3pt minus1pt +\def\small{\@setfontsize{\small}{8.5}{10pt}} +\def\footnotesize{\@setfontsize{\footnotesize}{8}{9pt}} +\def\scriptsize{\@setfontsize{\scriptsize}{7}{8pt}} +\def\tiny{\@setfontsize{\tiny}{5}{6pt}} +% sublargesize is the same as large - 10pt +\def\sublargesize{\@setfontsize{\sublargesize}{10}{12pt}} +\def\large{\@setfontsize{\large}{10}{12pt}} +\def\Large{\@setfontsize{\Large}{12}{14pt}} +\def\LARGE{\@setfontsize{\LARGE}{14}{17pt}} +\def\huge{\@setfontsize{\huge}{17}{20pt}} +\def\Huge{\@setfontsize{\Huge}{20}{24pt}} +\fi + + +% Check if we have selected 10 points +\ifx\CLASSOPTIONpt\@IEEEptsizeten +\typeout{-- This is a 10 point document.} +\def\normalsize{\@setfontsize{\normalsize}{10}{11}}% +\setlength{\@IEEEnormalsizeunitybaselineskip}{11pt}% +\normalsize +\abovedisplayskip 1.5ex plus4pt minus2pt% +\belowdisplayskip \abovedisplayskip% +\abovedisplayshortskip 0pt plus4pt% +\belowdisplayshortskip 1.5ex plus4pt minus2pt +\def\small{\@setfontsize{\small}{9}{10pt}} +\def\footnotesize{\@setfontsize{\footnotesize}{8}{9pt}} +\def\scriptsize{\@setfontsize{\scriptsize}{7}{8pt}} +\def\tiny{\@setfontsize{\tiny}{5}{6pt}} +% sublargesize is a tad smaller than large - 11pt +\def\sublargesize{\@setfontsize{\sublargesize}{11}{13.4pt}} +\def\large{\@setfontsize{\large}{12}{14pt}} +\def\Large{\@setfontsize{\Large}{14}{17pt}} +\def\LARGE{\@setfontsize{\LARGE}{17}{20pt}} +\def\huge{\@setfontsize{\huge}{20}{24pt}} +\def\Huge{\@setfontsize{\Huge}{24}{28pt}} +\fi + + +% Check if we have selected 11 points +\ifx\CLASSOPTIONpt\@IEEEptsizeeleven +\typeout{-- This is an 11 point document.} +\def\normalsize{\@setfontsize{\normalsize}{11}{13.3846pt}}% +\setlength{\@IEEEnormalsizeunitybaselineskip}{13.3846pt}% +\normalsize +\abovedisplayskip 1.5ex plus5pt minus3pt% +\belowdisplayskip \abovedisplayskip% +\abovedisplayshortskip 0pt plus5pt% +\belowdisplayshortskip 1.5ex plus5pt minus3pt +\def\small{\@setfontsize{\small}{10}{12pt}} +\def\footnotesize{\@setfontsize{\footnotesize}{9}{10.5pt}} +\def\scriptsize{\@setfontsize{\scriptsize}{8}{9pt}} +\def\tiny{\@setfontsize{\tiny}{6}{7pt}} +% sublargesize is the same as large - 12pt +\def\sublargesize{\@setfontsize{\sublargesize}{12}{14pt}} +\def\large{\@setfontsize{\large}{12}{14pt}} +\def\Large{\@setfontsize{\Large}{14}{17pt}} +\def\LARGE{\@setfontsize{\LARGE}{17}{20pt}} +\def\huge{\@setfontsize{\huge}{20}{24pt}} +\def\Huge{\@setfontsize{\Huge}{24}{28pt}} +\fi + + +% Check if we have selected 12 points +\ifx\CLASSOPTIONpt\@IEEEptsizetwelve +\typeout{-- This is a 12 point document.} +\def\normalsize{\@setfontsize{\normalsize}{12}{13.92pt}}% +\setlength{\@IEEEnormalsizeunitybaselineskip}{13.92pt}% +\normalsize +\abovedisplayskip 1.5ex plus6pt minus4pt% +\belowdisplayskip \abovedisplayskip% +\abovedisplayshortskip 0pt plus6pt% +\belowdisplayshortskip 1.5ex plus6pt minus4pt +\def\small{\@setfontsize{\small}{10}{12pt}} +\def\footnotesize{\@setfontsize{\footnotesize}{9}{10.5pt}} +\def\scriptsize{\@setfontsize{\scriptsize}{8}{9pt}} +\def\tiny{\@setfontsize{\tiny}{6}{7pt}} +% sublargesize is the same as large - 14pt +\def\sublargesize{\@setfontsize{\sublargesize}{14}{17pt}} +\def\large{\@setfontsize{\large}{14}{17pt}} +\def\Large{\@setfontsize{\Large}{17}{20pt}} +\def\LARGE{\@setfontsize{\LARGE}{20}{24pt}} +\def\huge{\@setfontsize{\huge}{22}{26pt}} +\def\Huge{\@setfontsize{\Huge}{24}{28pt}} +\fi + + +% V1.6 The Computer Modern Fonts will issue a substitution warning for +% 24pt titles (24.88pt is used instead) increase the substitution +% tolerance to turn off this warning +\def\fontsubfuzz{.9pt} +% However, the default (and correct) Times font will scale exactly as needed. + + +% warn the user in case they forget to use the 9pt option with +% technote +\ifCLASSOPTIONtechnote% + \ifx\CLASSOPTIONpt\@IEEEptsizenine\else% + \typeout{** ATTENTION: Technotes are normally 9pt documents.}% + \fi% +\fi + + +% V1.7 +% Improved \textunderscore to provide a much better fake _ when used with +% OT1 encoding. Under OT1, detect use of pcr or cmtt \ttfamily and use +% available true _ glyph for those two typewriter fonts. +\def\@IEEEstringptm{ptm} % Times Roman family +\def\@IEEEstringppl{ppl} % Palatino Roman family +\def\@IEEEstringphv{phv} % Helvetica Sans Serif family +\def\@IEEEstringpcr{pcr} % Courier typewriter family +\def\@IEEEstringcmtt{cmtt} % Computer Modern typewriter family +\DeclareTextCommandDefault{\textunderscore}{\leavevmode +\ifx\f@family\@IEEEstringpcr\string_\else +\ifx\f@family\@IEEEstringcmtt\string_\else +\ifx\f@family\@IEEEstringptm\kern 0em\vbox{\hrule\@width 0.5em\@height 0.5pt\kern -0.3ex}\else +\ifx\f@family\@IEEEstringppl\kern 0em\vbox{\hrule\@width 0.5em\@height 0.5pt\kern -0.3ex}\else +\ifx\f@family\@IEEEstringphv\kern -0.03em\vbox{\hrule\@width 0.62em\@height 0.52pt\kern -0.33ex}\kern -0.03em\else +\kern 0.09em\vbox{\hrule\@width 0.6em\@height 0.44pt\kern -0.63pt\kern -0.42ex}\kern 0.09em\fi\fi\fi\fi\fi\relax} + + + + +% set the default \baselinestretch +\def\baselinestretch{1} +\ifCLASSOPTIONdraftcls + \def\baselinestretch{1.5}% default baselinestretch for draft modes +\fi + + +% process CLASSINPUT baselinestretch +\ifx\CLASSINPUTbaselinestretch\@IEEEundefined +\else + \edef\baselinestretch{\CLASSINPUTbaselinestretch} % user CLASSINPUT override + \typeout{** ATTENTION: Overriding \string\baselinestretch\space to + \baselinestretch\space via \string\CLASSINPUT.} +\fi + +\normalsize % make \baselinestretch take affect + + + + +% store the normalsize baselineskip +\newdimen\CLASSINFOnormalsizebaselineskip +\CLASSINFOnormalsizebaselineskip=\baselineskip\relax +% and the normalsize unity (baselinestretch=1) baselineskip +% we could save a register by giving the user access to +% \@IEEEnormalsizeunitybaselineskip. However, let's protect +% its read only internal status +\newdimen\CLASSINFOnormalsizeunitybaselineskip +\CLASSINFOnormalsizeunitybaselineskip=\@IEEEnormalsizeunitybaselineskip\relax +% store the nominal value of jot +\newdimen\IEEEnormaljot +\IEEEnormaljot=0.25\baselineskip\relax + +% set \jot +\jot=\IEEEnormaljot\relax + + + + +% V1.6, we are now going to fine tune the interword spacing +% The default interword glue for Times under TeX appears to use a +% nominal interword spacing of 25% (relative to the font size, i.e., 1em) +% a maximum of 40% and a minimum of 19%. +% For example, 10pt text uses an interword glue of: +% +% 2.5pt plus 1.49998pt minus 0.59998pt +% +% However, IEEE allows for a more generous range which reduces the need +% for hyphenation, especially for two column text. Furthermore, IEEE +% tends to use a little bit more nominal space between the words. +% IEEE's interword spacing percentages appear to be: +% 35% nominal +% 23% minimum +% 50% maximum +% (They may even be using a tad more for the largest fonts such as 24pt.) +% +% for bold text, IEEE increases the spacing a little more: +% 37.5% nominal +% 23% minimum +% 55% maximum + +% here are the interword spacing ratios we'll use +% for medium (normal weight) +\def\@IEEEinterspaceratioM{0.35} +\def\@IEEEinterspaceMINratioM{0.23} +\def\@IEEEinterspaceMAXratioM{0.50} + +% for bold +\def\@IEEEinterspaceratioB{0.375} +\def\@IEEEinterspaceMINratioB{0.23} +\def\@IEEEinterspaceMAXratioB{0.55} + + +% command to revise the interword spacing for the current font under TeX: +% \fontdimen2 = nominal interword space +% \fontdimen3 = interword stretch +% \fontdimen4 = interword shrink +% since all changes to the \fontdimen are global, we can enclose these commands +% in braces to confine any font attribute or length changes +\def\@@@IEEEsetfontdimens#1#2#3{{% +\setlength{\@IEEEtrantmpdimenB}{\f@size pt}% grab the font size in pt, could use 1em instead. +\setlength{\@IEEEtrantmpdimenA}{#1\@IEEEtrantmpdimenB}% +\fontdimen2\font=\@IEEEtrantmpdimenA\relax +\addtolength{\@IEEEtrantmpdimenA}{-#2\@IEEEtrantmpdimenB}% +\fontdimen3\font=-\@IEEEtrantmpdimenA\relax +\setlength{\@IEEEtrantmpdimenA}{#1\@IEEEtrantmpdimenB}% +\addtolength{\@IEEEtrantmpdimenA}{-#3\@IEEEtrantmpdimenB}% +\fontdimen4\font=\@IEEEtrantmpdimenA\relax}} + +% revise the interword spacing for each font weight +\def\@@IEEEsetfontdimens{{% +\mdseries +\@@@IEEEsetfontdimens{\@IEEEinterspaceratioM}{\@IEEEinterspaceMAXratioM}{\@IEEEinterspaceMINratioM}% +\bfseries +\@@@IEEEsetfontdimens{\@IEEEinterspaceratioB}{\@IEEEinterspaceMAXratioB}{\@IEEEinterspaceMINratioB}% +}} + +% revise the interword spacing for each font shape +% \slshape is not often used for IEEE work and is not altered here. The \scshape caps are +% already a tad too large in the free LaTeX fonts (as compared to what IEEE uses) so we +% won't alter these either. +\def\@IEEEsetfontdimens{{% +\normalfont +\@@IEEEsetfontdimens +\normalfont\itshape +\@@IEEEsetfontdimens +}} + +% command to revise the interword spacing for each font size (and shape +% and weight). Only the \rmfamily is done here as \ttfamily uses a +% fixed spacing and \sffamily is not used as the main text of IEEE papers. +\def\@IEEEtunefonts{{\selectfont\rmfamily +\tiny\@IEEEsetfontdimens +\scriptsize\@IEEEsetfontdimens +\footnotesize\@IEEEsetfontdimens +\small\@IEEEsetfontdimens +\normalsize\@IEEEsetfontdimens +\sublargesize\@IEEEsetfontdimens +\large\@IEEEsetfontdimens +\LARGE\@IEEEsetfontdimens +\huge\@IEEEsetfontdimens +\Huge\@IEEEsetfontdimens}} + +% if the nofonttune class option is not given, revise the interword spacing +% now - in case IEEEtran makes any default length measurements, and make +% sure all the default fonts are loaded +\ifCLASSOPTIONnofonttune\else +\@IEEEtunefonts +\fi + +% and again at the start of the document in case the user loaded different fonts +\AtBeginDocument{\ifCLASSOPTIONnofonttune\else\@IEEEtunefonts\fi} + + + +% V1.6 +% LaTeX is a little to quick to use hyphenations +% So, we increase the penalty for their use and raise +% the badness level that triggers an underfull hbox +% warning. The author may still have to tweak things, +% but the appearance will be much better "right out +% of the box" than that under V1.5 and prior. +% TeX default is 50 +\hyphenpenalty=750 +% If we didn't adjust the interword spacing, 2200 might be better. +% The TeX default is 1000 +\hbadness=1350 +% IEEE does not use extra spacing after punctuation +\frenchspacing + +% V1.7 increase this a tad to discourage equation breaks +\binoppenalty=1000 % default 700 +\relpenalty=800 % default 500 + + +% margin note stuff +\marginparsep 10pt +\marginparwidth 20pt +\marginparpush 25pt + + +% if things get too close, go ahead and let them touch +\lineskip 0pt +\normallineskip 0pt +\lineskiplimit 0pt +\normallineskiplimit 0pt + +% The distance from the lower edge of the text body to the +% footline +\footskip 0.4in + +% normally zero, should be relative to font height. +% put in a little rubber to help stop some bad breaks (underfull vboxes) +\parskip 0ex plus 0.2ex minus 0.1ex +\ifCLASSOPTIONconference +\parskip 6pt plus 2pt minus 1pt +\fi + +\parindent 1.0em +\ifCLASSOPTIONconference +\parindent 14.45pt +\fi + +\topmargin -49.0pt +\headheight 12pt +\headsep 0.25in + +% use the normal font baselineskip +% so that \topskip is unaffected by changes in \baselinestretch +\topskip=\@IEEEnormalsizeunitybaselineskip +\textheight 58pc % 9.63in, 696pt +% Tweak textheight to a perfect integer number of lines/page. +% The normal baselineskip for each document point size is used +% to determine these values. +\ifx\CLASSOPTIONpt\@IEEEptsizenine\textheight=63\@IEEEnormalsizeunitybaselineskip\fi % 63 lines/page +\ifx\CLASSOPTIONpt\@IEEEptsizeten\textheight=58\@IEEEnormalsizeunitybaselineskip\fi % 58 lines/page +\ifx\CLASSOPTIONpt\@IEEEptsizeeleven\textheight=52\@IEEEnormalsizeunitybaselineskip\fi % 52 lines/page +\ifx\CLASSOPTIONpt\@IEEEptsizetwelve\textheight=50\@IEEEnormalsizeunitybaselineskip\fi % 50 lines/page + + +\columnsep 1.5pc +\textwidth 184.2mm + + +% the default side margins are equal +\if@IEEEusingAfourpaper +\oddsidemargin 14.32mm +\evensidemargin 14.32mm +\else +\oddsidemargin 0.680in +\evensidemargin 0.680in +\fi +% compensate for LaTeX's 1in offset +\addtolength{\oddsidemargin}{-1in} +\addtolength{\evensidemargin}{-1in} + + + +% adjust margins for conference mode +\ifCLASSOPTIONconference + \topmargin -0.25in + % we retain the reserved, but unused space for headers + \addtolength{\topmargin}{-\headheight} + \addtolength{\topmargin}{-\headsep} + \textheight 9.25in % The standard for conferences (668.4975pt) + % Tweak textheight to a perfect integer number of lines/page. + \ifx\CLASSOPTIONpt\@IEEEptsizenine\textheight=61\@IEEEnormalsizeunitybaselineskip\fi % 61 lines/page + \ifx\CLASSOPTIONpt\@IEEEptsizeten\textheight=62\@IEEEnormalsizeunitybaselineskip\fi % 62 lines/page + \ifx\CLASSOPTIONpt\@IEEEptsizeeleven\textheight=50\@IEEEnormalsizeunitybaselineskip\fi % 50 lines/page + \ifx\CLASSOPTIONpt\@IEEEptsizetwelve\textheight=48\@IEEEnormalsizeunitybaselineskip\fi % 48 lines/page +\fi + + +% compsoc conference +\ifCLASSOPTIONcompsoc +\ifCLASSOPTIONconference + % compsoc conference use a larger value for columnsep + \columnsep 0.375in + % compsoc conferences want 1in top margin, 1.125in bottom margin + \topmargin 0in + \addtolength{\topmargin}{-6pt}% we tweak this a tad to better comply with top of line stuff + % we retain the reserved, but unused space for headers + \addtolength{\topmargin}{-\headheight} + \addtolength{\topmargin}{-\headsep} + \textheight 8.875in % (641.39625pt) + % Tweak textheight to a perfect integer number of lines/page. + \ifx\CLASSOPTIONpt\@IEEEptsizenine\textheight=58\@IEEEnormalsizeunitybaselineskip\fi % 58 lines/page + \ifx\CLASSOPTIONpt\@IEEEptsizeten\textheight=53\@IEEEnormalsizeunitybaselineskip\fi % 53 lines/page + \ifx\CLASSOPTIONpt\@IEEEptsizeeleven\textheight=48\@IEEEnormalsizeunitybaselineskip\fi % 48 lines/page + \ifx\CLASSOPTIONpt\@IEEEptsizetwelve\textheight=46\@IEEEnormalsizeunitybaselineskip\fi % 46 lines/page + \textwidth 6.5in + % the default side margins are equal + \if@IEEEusingAfourpaper + \oddsidemargin 22.45mm + \evensidemargin 22.45mm + \else + \oddsidemargin 1in + \evensidemargin 1in + \fi + % compensate for LaTeX's 1in offset + \addtolength{\oddsidemargin}{-1in} + \addtolength{\evensidemargin}{-1in} +\fi\fi + + + +% draft mode settings override that of all other modes +% provides a nice 1in margin all around the paper and extra +% space between the lines for editor's comments +\ifCLASSOPTIONdraftcls + % want 1in from top of paper to text + \setlength{\topmargin}{-\headsep}% + \addtolength{\topmargin}{-\headheight}% + % we want 1in side margins regardless of paper type + \oddsidemargin 0in + \evensidemargin 0in + % set the text width + \setlength{\textwidth}{\paperwidth}% + \addtolength{\textwidth}{-2.0in}% + \setlength{\textheight}{\paperheight}% + \addtolength{\textheight}{-2.0in}% + % digitize textheight to be an integer number of lines. + % this may cause the bottom margin to be off a tad + \addtolength{\textheight}{-1\topskip}% + \divide\textheight by \baselineskip% + \multiply\textheight by \baselineskip% + \addtolength{\textheight}{\topskip}% +\fi + + + +% process CLASSINPUT inner/outer margin +% if inner margin defined, but outer margin not, set outer to inner. +\ifx\CLASSINPUTinnersidemargin\@IEEEundefined +\else + \ifx\CLASSINPUToutersidemargin\@IEEEundefined + \edef\CLASSINPUToutersidemargin{\CLASSINPUTinnersidemargin} + \fi +\fi + +\ifx\CLASSINPUToutersidemargin\@IEEEundefined +\else + % if outer margin defined, but inner margin not, set inner to outer. + \ifx\CLASSINPUTinnersidemargin\@IEEEundefined + \edef\CLASSINPUTinnersidemargin{\CLASSINPUToutersidemargin} + \fi + \setlength{\oddsidemargin}{\CLASSINPUTinnersidemargin} + \ifCLASSOPTIONtwoside + \setlength{\evensidemargin}{\CLASSINPUToutersidemargin} + \else + \setlength{\evensidemargin}{\CLASSINPUTinnersidemargin} + \fi + \addtolength{\oddsidemargin}{-1in} + \addtolength{\evensidemargin}{-1in} + \setlength{\textwidth}{\paperwidth} + \addtolength{\textwidth}{-\CLASSINPUTinnersidemargin} + \addtolength{\textwidth}{-\CLASSINPUToutersidemargin} + \typeout{** ATTENTION: Overriding inner side margin to \CLASSINPUTinnersidemargin\space and + outer side margin to \CLASSINPUToutersidemargin\space via \string\CLASSINPUT.} +\fi + + + +% process CLASSINPUT top/bottom text margin +% if toptext margin defined, but bottomtext margin not, set bottomtext to toptext margin +\ifx\CLASSINPUTtoptextmargin\@IEEEundefined +\else + \ifx\CLASSINPUTbottomtextmargin\@IEEEundefined + \edef\CLASSINPUTbottomtextmargin{\CLASSINPUTtoptextmargin} + \fi +\fi + +\ifx\CLASSINPUTbottomtextmargin\@IEEEundefined +\else + % if bottomtext margin defined, but toptext margin not, set toptext to bottomtext margin + \ifx\CLASSINPUTtoptextmargin\@IEEEundefined + \edef\CLASSINPUTtoptextmargin{\CLASSINPUTbottomtextmargin} + \fi + \setlength{\topmargin}{\CLASSINPUTtoptextmargin} + \addtolength{\topmargin}{-1in} + \addtolength{\topmargin}{-\headheight} + \addtolength{\topmargin}{-\headsep} + \setlength{\textheight}{\paperheight} + \addtolength{\textheight}{-\CLASSINPUTtoptextmargin} + \addtolength{\textheight}{-\CLASSINPUTbottomtextmargin} + % in the default format we use the normal baselineskip as topskip + % we only need 0.7 of this to clear typical top text and we need + % an extra 0.3 spacing at the bottom for descenders. This will + % correct for both. + \addtolength{\topmargin}{-0.3\@IEEEnormalsizeunitybaselineskip} + \typeout{** ATTENTION: Overriding top text margin to \CLASSINPUTtoptextmargin\space and + bottom text margin to \CLASSINPUTbottomtextmargin\space via \string\CLASSINPUT.} +\fi + + + + + + + +% LIST SPACING CONTROLS + +% Controls the amount of EXTRA spacing +% above and below \trivlist +% Both \list and IED lists override this. +% However, \trivlist will use this as will most +% things built from \trivlist like the \center +% environment. +\topsep 0.5\baselineskip + +% Controls the additional spacing around lists preceded +% or followed by blank lines. IEEE does not increase +% spacing before or after paragraphs so it is set to zero. +% \z@ is the same as zero, but faster. +\partopsep \z@ + +% Controls the spacing between paragraphs in lists. +% IEEE does not increase spacing before or after paragraphs +% so this is also zero. +% With IEEEtran.cls, global changes to +% this value DO affect lists (but not IED lists). +\parsep \z@ + +% Controls the extra spacing between list items. +% IEEE does not put extra spacing between items. +% With IEEEtran.cls, global changes to this value DO affect +% lists (but not IED lists). +\itemsep \z@ + +% \itemindent is the amount to indent the FIRST line of a list +% item. It is auto set to zero within the \list environment. To alter +% it, you have to do so when you call the \list. +% However, IEEE uses this for the theorem environment +% There is an alternative value for this near \leftmargini below +\itemindent -1em + +% \leftmargin, the spacing from the left margin of the main text to +% the left of the main body of a list item is set by \list. +% Hence this statement does nothing for lists. +% But, quote and verse do use it for indention. +\leftmargin 2em + +% we retain this stuff from the older IEEEtran.cls so that \list +% will work the same way as before. However, itemize, enumerate and +% description (IED) could care less about what these are as they +% all are overridden. +\leftmargini 2em +%\itemindent 2em % Alternative values: sometimes used. +%\leftmargini 0em +\leftmarginii 1em +\leftmarginiii 1.5em +\leftmarginiv 1.5em +\leftmarginv 1.0em +\leftmarginvi 1.0em +\labelsep 0.5em +\labelwidth \z@ + + +% The old IEEEtran.cls behavior of \list is retained. +% However, the new V1.3 IED list environments override all the +% @list stuff (\@listX is called within \list for the +% appropriate level just before the user's list_decl is called). +% \topsep is now 2pt as IEEE puts a little extra space around +% lists - used by those non-IED macros that depend on \list. +% Note that \parsep and \itemsep are not redefined as in +% the sizexx.clo \@listX (which article.cls uses) so global changes +% of these values DO affect \list +% +\def\@listi{\leftmargin\leftmargini \topsep 2pt plus 1pt minus 1pt} +\let\@listI\@listi +\def\@listii{\leftmargin\leftmarginii\labelwidth\leftmarginii% + \advance\labelwidth-\labelsep \topsep 2pt} +\def\@listiii{\leftmargin\leftmarginiii\labelwidth\leftmarginiii% + \advance\labelwidth-\labelsep \topsep 2pt} +\def\@listiv{\leftmargin\leftmarginiv\labelwidth\leftmarginiv% + \advance\labelwidth-\labelsep \topsep 2pt} +\def\@listv{\leftmargin\leftmarginv\labelwidth\leftmarginv% + \advance\labelwidth-\labelsep \topsep 2pt} +\def\@listvi{\leftmargin\leftmarginvi\labelwidth\leftmarginvi% + \advance\labelwidth-\labelsep \topsep 2pt} + + +% IEEE uses 5) not 5. +\def\labelenumi{\theenumi)} \def\theenumi{\arabic{enumi}} + +% IEEE uses a) not (a) +\def\labelenumii{\theenumii)} \def\theenumii{\alph{enumii}} + +% IEEE uses iii) not iii. +\def\labelenumiii{\theenumiii)} \def\theenumiii{\roman{enumiii}} + +% IEEE uses A) not A. +\def\labelenumiv{\theenumiv)} \def\theenumiv{\Alph{enumiv}} + +% exactly the same as in article.cls +\def\p@enumii{\theenumi} +\def\p@enumiii{\theenumi(\theenumii)} +\def\p@enumiv{\p@enumiii\theenumiii} + +% itemized list label styles +\def\labelitemi{$\bullet$} +\def\labelitemii{$\circ$} +\def\labelitemiii{\vrule height 0.8ex depth -0.2ex width 0.6ex} +\def\labelitemiv{$\ast$} + + + +% **** V1.3 ENHANCEMENTS **** +% Itemize, Enumerate and Description (IED) List Controls +% *************************** +% +% +% IEEE seems to use at least two different values by +% which ITEMIZED list labels are indented to the right +% For The Journal of Lightwave Technology (JLT) and The Journal +% on Selected Areas in Communications (JSAC), they tend to use +% an indention equal to \parindent. For Transactions on Communications +% they tend to indent ITEMIZED lists a little more--- 1.3\parindent. +% We'll provide both values here for you so that you can choose +% which one you like in your document using a command such as: +% setlength{\IEEEilabelindent}{\IEEEilabelindentB} +\newdimen\IEEEilabelindentA +\IEEEilabelindentA \parindent + +\newdimen\IEEEilabelindentB +\IEEEilabelindentB 1.3\parindent +% However, we'll default to using \parindent +% which makes more sense to me +\newdimen\IEEEilabelindent +\IEEEilabelindent \IEEEilabelindentA + + +% This controls the default amount the enumerated list labels +% are indented to the right. +% Normally, this is the same as the paragraph indention +\newdimen\IEEEelabelindent +\IEEEelabelindent \parindent + +% This controls the default amount the description list labels +% are indented to the right. +% Normally, this is the same as the paragraph indention +\newdimen\IEEEdlabelindent +\IEEEdlabelindent \parindent + +% This is the value actually used within the IED lists. +% The IED environments automatically set its value to +% one of the three values above, so global changes do +% not have any effect +\newdimen\IEEElabelindent +\IEEElabelindent \parindent + +% The actual amount labels will be indented is +% \IEEElabelindent multiplied by the factor below +% corresponding to the level of nesting depth +% This provides a means by which the user can +% alter the effective \IEEElabelindent for deeper +% levels +% There may not be such a thing as correct "standard IEEE" +% values. What IEEE actually does may depend on the specific +% circumstances. +% The first list level almost always has full indention. +% The second levels I've seen have only 75% of the normal indentation +% Three level or greater nestings are very rare. I am guessing +% that they don't use any indentation. +\def\IEEElabelindentfactori{1.0} % almost always one +\def\IEEElabelindentfactorii{0.75} % 0.0 or 1.0 may be used in some cases +\def\IEEElabelindentfactoriii{0.0} % 0.75? 0.5? 0.0? +\def\IEEElabelindentfactoriv{0.0} +\def\IEEElabelindentfactorv{0.0} +\def\IEEElabelindentfactorvi{0.0} + +% value actually used within IED lists, it is auto +% set to one of the 6 values above +% global changes here have no effect +\def\IEEElabelindentfactor{1.0} + +% This controls the default spacing between the end of the IED +% list labels and the list text, when normal text is used for +% the labels. +\newdimen\IEEEiednormlabelsep +\IEEEiednormlabelsep \parindent + +% This controls the default spacing between the end of the IED +% list labels and the list text, when math symbols are used for +% the labels (nomenclature lists). IEEE usually increases the +% spacing in these cases +\newdimen\IEEEiedmathlabelsep +\IEEEiedmathlabelsep 1.2em + +% This controls the extra vertical separation put above and +% below each IED list. IEEE usually puts a little extra spacing +% around each list. However, this spacing is barely noticeable. +\newskip\IEEEiedtopsep +\IEEEiedtopsep 2pt plus 1pt minus 1pt + + +% This command is executed within each IED list environment +% at the beginning of the list. You can use this to set the +% parameters for some/all your IED list(s) without disturbing +% global parameters that affect things other than lists. +% i.e., renewcommand{\IEEEiedlistdecl}{\setlength{\labelsep}{5em}} +% will alter the \labelsep for the next list(s) until +% \IEEEiedlistdecl is redefined. +\def\IEEEiedlistdecl{\relax} + +% This command provides an easy way to set \leftmargin based +% on the \labelwidth, \labelsep and the argument \IEEElabelindent +% Usage: \IEEEcalcleftmargin{width-to-indent-the-label} +% output is in the \leftmargin variable, i.e., effectively: +% \leftmargin = argument + \labelwidth + \labelsep +% Note controlled spacing here, shield end of lines with % +\def\IEEEcalcleftmargin#1{\setlength{\leftmargin}{#1}% +\addtolength{\leftmargin}{\labelwidth}% +\addtolength{\leftmargin}{\labelsep}} + +% This command provides an easy way to set \labelwidth to the +% width of the given text. It is the same as +% \settowidth{\labelwidth}{label-text} +% and useful as a shorter alternative. +% Typically used to set \labelwidth to be the width +% of the longest label in the list +\def\IEEEsetlabelwidth#1{\settowidth{\labelwidth}{#1}} + +% When this command is executed, IED lists will use the +% IEEEiedmathlabelsep label separation rather than the normal +% spacing. To have an effect, this command must be executed via +% the \IEEEiedlistdecl or within the option of the IED list +% environments. +\def\IEEEusemathlabelsep{\setlength{\labelsep}{\IEEEiedmathlabelsep}} + +% A flag which controls whether the IED lists automatically +% calculate \leftmargin from \IEEElabelindent, \labelwidth and \labelsep +% Useful if you want to specify your own \leftmargin +% This flag must be set (\IEEEnocalcleftmargintrue or \IEEEnocalcleftmarginfalse) +% via the \IEEEiedlistdecl or within the option of the IED list +% environments to have an effect. +\newif\ifIEEEnocalcleftmargin +\IEEEnocalcleftmarginfalse + +% A flag which controls whether \IEEElabelindent is multiplied by +% the \IEEElabelindentfactor for each list level. +% This flag must be set via the \IEEEiedlistdecl or within the option +% of the IED list environments to have an effect. +\newif\ifIEEEnolabelindentfactor +\IEEEnolabelindentfactorfalse + + +% internal variable to indicate type of IED label +% justification +% 0 - left; 1 - center; 2 - right +\def\@IEEEiedjustify{0} + + +% commands to allow the user to control IED +% label justifications. Use these commands within +% the IED environment option or in the \IEEEiedlistdecl +% Note that changing the normal list justifications +% is nonstandard and IEEE may not like it if you do so! +% I include these commands as they may be helpful to +% those who are using these enhanced list controls for +% other non-IEEE related LaTeX work. +% itemize and enumerate automatically default to right +% justification, description defaults to left. +\def\IEEEiedlabeljustifyl{\def\@IEEEiedjustify{0}}%left +\def\IEEEiedlabeljustifyc{\def\@IEEEiedjustify{1}}%center +\def\IEEEiedlabeljustifyr{\def\@IEEEiedjustify{2}}%right + + + + +% commands to save to and restore from the list parameter copies +% this allows us to set all the list parameters within +% the list_decl and prevent \list (and its \@list) +% from overriding any of our parameters +% V1.6 use \edefs instead of dimen's to conserve dimen registers +% Note controlled spacing here, shield end of lines with % +\def\@IEEEsavelistparams{\edef\@IEEEiedtopsep{\the\topsep}% +\edef\@IEEEiedlabelwidth{\the\labelwidth}% +\edef\@IEEEiedlabelsep{\the\labelsep}% +\edef\@IEEEiedleftmargin{\the\leftmargin}% +\edef\@IEEEiedpartopsep{\the\partopsep}% +\edef\@IEEEiedparsep{\the\parsep}% +\edef\@IEEEieditemsep{\the\itemsep}% +\edef\@IEEEiedrightmargin{\the\rightmargin}% +\edef\@IEEEiedlistparindent{\the\listparindent}% +\edef\@IEEEieditemindent{\the\itemindent}} + +% Note controlled spacing here +\def\@IEEErestorelistparams{\topsep\@IEEEiedtopsep\relax% +\labelwidth\@IEEEiedlabelwidth\relax% +\labelsep\@IEEEiedlabelsep\relax% +\leftmargin\@IEEEiedleftmargin\relax% +\partopsep\@IEEEiedpartopsep\relax% +\parsep\@IEEEiedparsep\relax% +\itemsep\@IEEEieditemsep\relax% +\rightmargin\@IEEEiedrightmargin\relax% +\listparindent\@IEEEiedlistparindent\relax% +\itemindent\@IEEEieditemindent\relax} + + +% v1.6b provide original LaTeX IED list environments +% note that latex.ltx defines \itemize and \enumerate, but not \description +% which must be created by the base classes +% save original LaTeX itemize and enumerate +\let\LaTeXitemize\itemize +\let\endLaTeXitemize\enditemize +\let\LaTeXenumerate\enumerate +\let\endLaTeXenumerate\endenumerate + +% provide original LaTeX description environment from article.cls +\newenvironment{LaTeXdescription} + {\list{}{\labelwidth\z@ \itemindent-\leftmargin + \let\makelabel\descriptionlabel}} + {\endlist} +\newcommand*\descriptionlabel[1]{\hspace\labelsep + \normalfont\bfseries #1} + + +% override LaTeX's default IED lists +\def\itemize{\@IEEEitemize} +\def\enditemize{\@endIEEEitemize} +\def\enumerate{\@IEEEenumerate} +\def\endenumerate{\@endIEEEenumerate} +\def\description{\@IEEEdescription} +\def\enddescription{\@endIEEEdescription} + +% provide the user with aliases - may help those using packages that +% override itemize, enumerate, or description +\def\IEEEitemize{\@IEEEitemize} +\def\endIEEEitemize{\@endIEEEitemize} +\def\IEEEenumerate{\@IEEEenumerate} +\def\endIEEEenumerate{\@endIEEEenumerate} +\def\IEEEdescription{\@IEEEdescription} +\def\endIEEEdescription{\@endIEEEdescription} + + +% V1.6 we want to keep the IEEEtran IED list definitions as our own internal +% commands so they are protected against redefinition +\def\@IEEEitemize{\@ifnextchar[{\@@IEEEitemize}{\@@IEEEitemize[\relax]}} +\def\@IEEEenumerate{\@ifnextchar[{\@@IEEEenumerate}{\@@IEEEenumerate[\relax]}} +\def\@IEEEdescription{\@ifnextchar[{\@@IEEEdescription}{\@@IEEEdescription[\relax]}} +\def\@endIEEEitemize{\endlist} +\def\@endIEEEenumerate{\endlist} +\def\@endIEEEdescription{\endlist} + + +% DO NOT ALLOW BLANK LINES TO BE IN THESE IED ENVIRONMENTS +% AS THIS WILL FORCE NEW PARAGRAPHS AFTER THE IED LISTS +% IEEEtran itemized list MDS 1/2001 +% Note controlled spacing here, shield end of lines with % +\def\@@IEEEitemize[#1]{% + \ifnum\@itemdepth>3\relax\@toodeep\else% + \ifnum\@listdepth>5\relax\@toodeep\else% + \advance\@itemdepth\@ne% + \edef\@itemitem{labelitem\romannumeral\the\@itemdepth}% + % get the labelindentfactor for this level + \advance\@listdepth\@ne% we need to know what the level WILL be + \edef\IEEElabelindentfactor{\csname IEEElabelindentfactor\romannumeral\the\@listdepth\endcsname}% + \advance\@listdepth-\@ne% undo our increment + \def\@IEEEiedjustify{2}% right justified labels are default + % set other defaults + \IEEEnocalcleftmarginfalse% + \IEEEnolabelindentfactorfalse% + \topsep\IEEEiedtopsep% + \IEEElabelindent\IEEEilabelindent% + \labelsep\IEEEiednormlabelsep% + \partopsep 0ex% + \parsep 0ex% + \itemsep \parskip% + \rightmargin 0em% + \listparindent 0em% + \itemindent 0em% + % calculate the label width + % the user can override this later if + % they specified a \labelwidth + \settowidth{\labelwidth}{\csname labelitem\romannumeral\the\@itemdepth\endcsname}% + \@IEEEsavelistparams% save our list parameters + \list{\csname\@itemitem\endcsname}{% + \@IEEErestorelistparams% override any list{} changes + % to our globals + \let\makelabel\@IEEEiedmakelabel% v1.6b setup \makelabel + \IEEEiedlistdecl% let user alter parameters + #1\relax% + % If the user has requested not to use the + % labelindent factor, don't revise \labelindent + \ifIEEEnolabelindentfactor\relax% + \else\IEEElabelindent=\IEEElabelindentfactor\labelindent% + \fi% + % Unless the user has requested otherwise, + % calculate our left margin based + % on \IEEElabelindent, \labelwidth and + % \labelsep + \ifIEEEnocalcleftmargin\relax% + \else\IEEEcalcleftmargin{\IEEElabelindent}% + \fi}\fi\fi}% + + +% DO NOT ALLOW BLANK LINES TO BE IN THESE IED ENVIRONMENTS +% AS THIS WILL FORCE NEW PARAGRAPHS AFTER THE IED LISTS +% IEEEtran enumerate list MDS 1/2001 +% Note controlled spacing here, shield end of lines with % +\def\@@IEEEenumerate[#1]{% + \ifnum\@enumdepth>3\relax\@toodeep\else% + \ifnum\@listdepth>5\relax\@toodeep\else% + \advance\@enumdepth\@ne% + \edef\@enumctr{enum\romannumeral\the\@enumdepth}% + % get the labelindentfactor for this level + \advance\@listdepth\@ne% we need to know what the level WILL be + \edef\IEEElabelindentfactor{\csname IEEElabelindentfactor\romannumeral\the\@listdepth\endcsname}% + \advance\@listdepth-\@ne% undo our increment + \def\@IEEEiedjustify{2}% right justified labels are default + % set other defaults + \IEEEnocalcleftmarginfalse% + \IEEEnolabelindentfactorfalse% + \topsep\IEEEiedtopsep% + \IEEElabelindent\IEEEelabelindent% + \labelsep\IEEEiednormlabelsep% + \partopsep 0ex% + \parsep 0ex% + \itemsep 0ex% + \rightmargin 0em% + \listparindent 0em% + \itemindent 0em% + % calculate the label width + % We'll set it to the width suitable for all labels using + % normalfont 1) to 9) + % The user can override this later + \settowidth{\labelwidth}{9)}% + \@IEEEsavelistparams% save our list parameters + \list{\csname label\@enumctr\endcsname}{\usecounter{\@enumctr}% + \@IEEErestorelistparams% override any list{} changes + % to our globals + \let\makelabel\@IEEEiedmakelabel% v1.6b setup \makelabel + \IEEEiedlistdecl% let user alter parameters + #1\relax% + % If the user has requested not to use the + % IEEElabelindent factor, don't revise \IEEElabelindent + \ifIEEEnolabelindentfactor\relax% + \else\IEEElabelindent=\IEEElabelindentfactor\IEEElabelindent% + \fi% + % Unless the user has requested otherwise, + % calculate our left margin based + % on \IEEElabelindent, \labelwidth and + % \labelsep + \ifIEEEnocalcleftmargin\relax% + \else\IEEEcalcleftmargin{\IEEElabelindent}% + \fi}\fi\fi}% + + +% DO NOT ALLOW BLANK LINES TO BE IN THESE IED ENVIRONMENTS +% AS THIS WILL FORCE NEW PARAGRAPHS AFTER THE IED LISTS +% IEEEtran description list MDS 1/2001 +% Note controlled spacing here, shield end of lines with % +\def\@@IEEEdescription[#1]{% + \ifnum\@listdepth>5\relax\@toodeep\else% + % get the labelindentfactor for this level + \advance\@listdepth\@ne% we need to know what the level WILL be + \edef\IEEElabelindentfactor{\csname IEEElabelindentfactor\romannumeral\the\@listdepth\endcsname}% + \advance\@listdepth-\@ne% undo our increment + \def\@IEEEiedjustify{0}% left justified labels are default + % set other defaults + \IEEEnocalcleftmarginfalse% + \IEEEnolabelindentfactorfalse% + \topsep\IEEEiedtopsep% + \IEEElabelindent\IEEEdlabelindent% + % assume normal labelsep + \labelsep\IEEEiednormlabelsep% + \partopsep 0ex% + \parsep 0ex% + \itemsep 0ex% + \rightmargin 0em% + \listparindent 0em% + \itemindent 0em% + % Bogus label width in case the user forgets + % to set it. + % TIP: If you want to see what a variable's width is you + % can use the TeX command \showthe\width-variable to + % display it on the screen during compilation + % (This might be helpful to know when you need to find out + % which label is the widest) + \settowidth{\labelwidth}{Hello}% + \@IEEEsavelistparams% save our list parameters + \list{}{\@IEEErestorelistparams% override any list{} changes + % to our globals + \let\makelabel\@IEEEiedmakelabel% v1.6b setup \makelabel + \IEEEiedlistdecl% let user alter parameters + #1\relax% + % If the user has requested not to use the + % labelindent factor, don't revise \IEEElabelindent + \ifIEEEnolabelindentfactor\relax% + \else\IEEElabelindent=\IEEElabelindentfactor\IEEElabelindent% + \fi% + % Unless the user has requested otherwise, + % calculate our left margin based + % on \IEEElabelindent, \labelwidth and + % \labelsep + \ifIEEEnocalcleftmargin\relax% + \else\IEEEcalcleftmargin{\IEEElabelindent}\relax% + \fi}\fi} + +% v1.6b we use one makelabel that does justification as needed. +\def\@IEEEiedmakelabel#1{\relax\if\@IEEEiedjustify 0\relax +\makebox[\labelwidth][l]{\normalfont #1}\else +\if\@IEEEiedjustify 1\relax +\makebox[\labelwidth][c]{\normalfont #1}\else +\makebox[\labelwidth][r]{\normalfont #1}\fi\fi} + + +% VERSE and QUOTE +% V1.7 define environments with newenvironment +\newenvironment{verse}{\let\\=\@centercr + \list{}{\itemsep\z@ \itemindent -1.5em \listparindent \itemindent + \rightmargin\leftmargin\advance\leftmargin 1.5em}\item\relax} + {\endlist} +\newenvironment{quotation}{\list{}{\listparindent 1.5em \itemindent\listparindent + \rightmargin\leftmargin \parsep 0pt plus 1pt}\item\relax} + {\endlist} +\newenvironment{quote}{\list{}{\rightmargin\leftmargin}\item\relax} + {\endlist} + + +% \titlepage +% provided only for backward compatibility. \maketitle is the correct +% way to create the title page. +\newif\if@restonecol +\def\titlepage{\@restonecolfalse\if@twocolumn\@restonecoltrue\onecolumn + \else \newpage \fi \thispagestyle{empty}\c@page\z@} +\def\endtitlepage{\if@restonecol\twocolumn \else \newpage \fi} + +% standard values from article.cls +\arraycolsep 5pt +\arrayrulewidth .4pt +\doublerulesep 2pt + +\tabcolsep 6pt +\tabbingsep 0.5em + + +%% FOOTNOTES +% +%\skip\footins 10pt plus 4pt minus 2pt +% V1.6 respond to changes in font size +% space added above the footnotes (if present) +\skip\footins 0.9\baselineskip plus 0.4\baselineskip minus 0.2\baselineskip + +% V1.6, we need to make \footnotesep responsive to changes +% in \baselineskip or strange spacings will result when in +% draft mode. Here is a little LaTeX secret - \footnotesep +% determines the height of an invisible strut that is placed +% *above* the baseline of footnotes after the first. Since +% LaTeX considers the space for characters to be 0.7/baselineskip +% above the baseline and 0.3/baselineskip below it, we need to +% use 0.7/baselineskip as a \footnotesep to maintain equal spacing +% between all the lines of the footnotes. IEEE often uses a tad +% more, so use 0.8\baselineskip. This slightly larger value also helps +% the text to clear the footnote marks. Note that \thanks in IEEEtran +% uses its own value of \footnotesep which is set in \maketitle. +{\footnotesize +\global\footnotesep 0.8\baselineskip} + +\def\unnumberedfootnote{\gdef\@thefnmark{\quad}\@footnotetext} + +\skip\@mpfootins 0.3\baselineskip +\fboxsep = 3pt +\fboxrule = .4pt +% V1.6 use 1em, then use LaTeX2e's \@makefnmark +% Note that IEEE normally *left* aligns the footnote marks, so we don't need +% box resizing tricks here. +%\long\def\@makefnmark{\scriptsize\normalfont\@thefnmark} +\long\def\@makefntext#1{\parindent 1em\indent\hbox{\@makefnmark}#1}% V1.6 use 1em +\long\def\@maketablefntext#1{\raggedleft\leavevmode\hbox{\@makefnmark}#1} +% V1.7 compsoc does not use superscipts for footnote marks +\ifCLASSOPTIONcompsoc +\def\@IEEEcompsocmakefnmark{\hbox{\normalfont\@thefnmark.\ }} +\long\def\@makefntext#1{\parindent 1em\indent\hbox{\@IEEEcompsocmakefnmark}#1} +\fi + +% IEEE does not use footnote rules. Or do they? +\def\footnoterule{\vskip-2pt \hrule height 0.6pt depth \z@ \vskip1.6pt\relax} +\toks@\expandafter{\@setminipage\let\footnoterule\relax\footnotesep\z@} +\edef\@setminipage{\the\toks@} + +% V1.7 for compsoc, IEEE uses a footnote rule only for \thanks. We devise a "one-shot" +% system to implement this. +\newif\if@IEEEenableoneshotfootnoterule +\@IEEEenableoneshotfootnoterulefalse +\ifCLASSOPTIONcompsoc +\def\footnoterule{\relax\if@IEEEenableoneshotfootnoterule +\kern-5pt +\hbox to \columnwidth{\hfill\vrule width 0.5\columnwidth height 0.4pt\hfill} +\kern4.6pt +\global\@IEEEenableoneshotfootnoterulefalse +\else +\relax +\fi} +\fi + +% V1.6 do not allow LaTeX to break a footnote across multiple pages +\interfootnotelinepenalty=10000 + +% V1.6 discourage breaks within equations +% Note that amsmath normally sets this to 10000, +% but LaTeX2e normally uses 100. +\interdisplaylinepenalty=2500 + +% default allows section depth up to /paragraph +\setcounter{secnumdepth}{4} + +% technotes do not allow /paragraph +\ifCLASSOPTIONtechnote + \setcounter{secnumdepth}{3} +\fi +% neither do compsoc conferences +\@IEEEcompsocconfonly{\setcounter{secnumdepth}{3}} + + +\newcounter{section} +\newcounter{subsection}[section] +\newcounter{subsubsection}[subsection] +\newcounter{paragraph}[subsubsection] + +% used only by IEEEtran's IEEEeqnarray as other packages may +% have their own, different, implementations +\newcounter{IEEEsubequation}[equation] + +% as shown when called by user from \ref, \label and in table of contents +\def\theequation{\arabic{equation}} % 1 +\def\theIEEEsubequation{\theequation\alph{IEEEsubequation}} % 1a (used only by IEEEtran's IEEEeqnarray) +\ifCLASSOPTIONcompsoc +% compsoc is all arabic +\def\thesection{\arabic{section}} +\def\thesubsection{\thesection.\arabic{subsection}} +\def\thesubsubsection{\thesubsection.\arabic{subsubsection}} +\def\theparagraph{\thesubsubsection.\arabic{paragraph}} +\else +\def\thesection{\Roman{section}} % I +% V1.7, \mbox prevents breaks around - +\def\thesubsection{\mbox{\thesection-\Alph{subsection}}} % I-A +% V1.7 use I-A1 format used by IEEE rather than I-A.1 +\def\thesubsubsection{\thesubsection\arabic{subsubsection}} % I-A1 +\def\theparagraph{\thesubsubsection\alph{paragraph}} % I-A1a +\fi + +% From Heiko Oberdiek. Because of the \mbox in \thesubsection, we need to +% tell hyperref to disable the \mbox command when making PDF bookmarks. +% This done already with hyperref.sty version 6.74o and later, but +% it will not hurt to do it here again for users of older versions. +\@ifundefined{pdfstringdefPreHook}{\let\pdfstringdefPreHook\@empty}{}% +\g@addto@macro\pdfstringdefPreHook{\let\mbox\relax} + + +% Main text forms (how shown in main text headings) +% V1.6, using \thesection in \thesectiondis allows changes +% in the former to automatically appear in the latter +\ifCLASSOPTIONcompsoc + \ifCLASSOPTIONconference% compsoc conference + \def\thesectiondis{\thesection.} + \def\thesubsectiondis{\thesectiondis\arabic{subsection}.} + \def\thesubsubsectiondis{\thesubsectiondis\arabic{subsubsection}.} + \def\theparagraphdis{\thesubsubsectiondis\arabic{paragraph}.} + \else% compsoc not conferencs + \def\thesectiondis{\thesection} + \def\thesubsectiondis{\thesectiondis.\arabic{subsection}} + \def\thesubsubsectiondis{\thesubsectiondis.\arabic{subsubsection}} + \def\theparagraphdis{\thesubsubsectiondis.\arabic{paragraph}} + \fi +\else% not compsoc + \def\thesectiondis{\thesection.} % I. + \def\thesubsectiondis{\Alph{subsection}.} % B. + \def\thesubsubsectiondis{\arabic{subsubsection})} % 3) + \def\theparagraphdis{\alph{paragraph})} % d) +\fi + +% just like LaTeX2e's \@eqnnum +\def\theequationdis{{\normalfont \normalcolor (\theequation)}}% (1) +% IEEEsubequation used only by IEEEtran's IEEEeqnarray +\def\theIEEEsubequationdis{{\normalfont \normalcolor (\theIEEEsubequation)}}% (1a) +% redirect LaTeX2e's equation number display and all that depend on +% it, through IEEEtran's \theequationdis +\def\@eqnnum{\theequationdis} + + + +% V1.7 provide string macros as article.cls does +\def\contentsname{Contents} +\def\listfigurename{List of Figures} +\def\listtablename{List of Tables} +\def\refname{References} +\def\indexname{Index} +\def\figurename{Fig.} +\def\tablename{TABLE} +\@IEEEcompsocconfonly{\def\figurename{Figure}\def\tablename{Table}} +\def\partname{Part} +\def\appendixname{Appendix} +\def\abstractname{Abstract} +% IEEE specific names +\def\IEEEkeywordsname{Keywords} +\def\IEEEproofname{Proof} + + +% LIST OF FIGURES AND TABLES AND TABLE OF CONTENTS +% +\def\@pnumwidth{1.55em} +\def\@tocrmarg{2.55em} +\def\@dotsep{4.5} +\setcounter{tocdepth}{3} + +% adjusted some spacings here so that section numbers will not easily +% collide with the section titles. +% VIII; VIII-A; and VIII-A.1 are usually the worst offenders. +% MDS 1/2001 +\def\tableofcontents{\section*{\contentsname}\@starttoc{toc}} +\def\l@section#1#2{\addpenalty{\@secpenalty}\addvspace{1.0em plus 1pt}% + \@tempdima 2.75em \begingroup \parindent \z@ \rightskip \@pnumwidth% + \parfillskip-\@pnumwidth {\bfseries\leavevmode #1}\hfil\hbox to\@pnumwidth{\hss #2}\par% + \endgroup} +% argument format #1:level, #2:labelindent,#3:labelsep +\def\l@subsection{\@dottedtocline{2}{2.75em}{3.75em}} +\def\l@subsubsection{\@dottedtocline{3}{6.5em}{4.5em}} +% must provide \l@ defs for ALL sublevels EVEN if tocdepth +% is such as they will not appear in the table of contents +% these defs are how TOC knows what level these things are! +\def\l@paragraph{\@dottedtocline{4}{6.5em}{5.5em}} +\def\l@subparagraph{\@dottedtocline{5}{6.5em}{6.5em}} +\def\listoffigures{\section*{\listfigurename}\@starttoc{lof}} +\def\l@figure{\@dottedtocline{1}{0em}{2.75em}} +\def\listoftables{\section*{\listtablename}\@starttoc{lot}} +\let\l@table\l@figure + + +%% Definitions for floats +%% +%% Normal Floats +\floatsep 1\baselineskip plus 0.2\baselineskip minus 0.2\baselineskip +\textfloatsep 1.7\baselineskip plus 0.2\baselineskip minus 0.4\baselineskip +\@fptop 0pt plus 1fil +\@fpsep 0.75\baselineskip plus 2fil +\@fpbot 0pt plus 1fil +\def\topfraction{0.9} +\def\bottomfraction{0.4} +\def\floatpagefraction{0.8} +% V1.7, let top floats approach 90% of page +\def\textfraction{0.1} + +%% Double Column Floats +\dblfloatsep 1\baselineskip plus 0.2\baselineskip minus 0.2\baselineskip + +\dbltextfloatsep 1.7\baselineskip plus 0.2\baselineskip minus 0.4\baselineskip +% Note that it would be nice if the rubber here actually worked in LaTeX2e. +% There is a long standing limitation in LaTeX, first discovered (to the best +% of my knowledge) by Alan Jeffrey in 1992. LaTeX ignores the stretchable +% portion of \dbltextfloatsep, and as a result, double column figures can and +% do result in an non-integer number of lines in the main text columns with +% underfull vbox errors as a consequence. A post to comp.text.tex +% by Donald Arseneau confirms that this had not yet been fixed in 1998. +% IEEEtran V1.6 will fix this problem for you in the titles, but it doesn't +% protect you from other double floats. Happy vspace'ing. + +\@dblfptop 0pt plus 1fil +\@dblfpsep 0.75\baselineskip plus 2fil +\@dblfpbot 0pt plus 1fil +\def\dbltopfraction{0.8} +\def\dblfloatpagefraction{0.8} +\setcounter{dbltopnumber}{4} + +\intextsep 1\baselineskip plus 0.2\baselineskip minus 0.2\baselineskip +\setcounter{topnumber}{2} +\setcounter{bottomnumber}{2} +\setcounter{totalnumber}{4} + + + +% article class provides these, we should too. +\newlength\abovecaptionskip +\newlength\belowcaptionskip +% but only \abovecaptionskip is used above figure captions and *below* table +% captions +\setlength\abovecaptionskip{0.65\baselineskip} +\setlength\belowcaptionskip{0.75\baselineskip} +% V1.6 create hooks in case the caption spacing ever needs to be +% overridden by a user +\def\@IEEEfigurecaptionsepspace{\vskip\abovecaptionskip\relax}% +\def\@IEEEtablecaptionsepspace{\vskip\belowcaptionskip\relax}% + + +% 1.6b revise caption system so that \@makecaption uses two arguments +% as with LaTeX2e. Otherwise, there will be problems when using hyperref. +\def\@IEEEtablestring{table} + +\ifCLASSOPTIONcompsoc +% V1.7 compsoc \@makecaption +\ifCLASSOPTIONconference% compsoc conference +\long\def\@makecaption#1#2{% +% test if is a for a figure or table +\ifx\@captype\@IEEEtablestring% +% if a table, do table caption +\normalsize\begin{center}{\normalfont\sffamily\normalsize {#1.}~ #2}\end{center}% +\@IEEEtablecaptionsepspace +% if not a table, format it as a figure +\else +\@IEEEfigurecaptionsepspace +\setbox\@tempboxa\hbox{\normalfont\sffamily\normalsize {#1.}~ #2}% +\ifdim \wd\@tempboxa >\hsize% +% if caption is longer than a line, let it wrap around +\setbox\@tempboxa\hbox{\normalfont\sffamily\normalsize {#1.}~ }% +\parbox[t]{\hsize}{\normalfont\sffamily\normalsize \noindent\unhbox\@tempboxa#2}% +% if caption is shorter than a line, center +\else% +\hbox to\hsize{\normalfont\sffamily\normalsize\hfil\box\@tempboxa\hfil}% +\fi\fi} +\else% nonconference compsoc +\long\def\@makecaption#1#2{% +% test if is a for a figure or table +\ifx\@captype\@IEEEtablestring% +% if a table, do table caption +\normalsize\begin{center}{\normalfont\sffamily\normalsize #1}\\{\normalfont\sffamily\normalsize #2}\end{center}% +\@IEEEtablecaptionsepspace +% if not a table, format it as a figure +\else +\@IEEEfigurecaptionsepspace +\setbox\@tempboxa\hbox{\normalfont\sffamily\normalsize {#1.}~ #2}% +\ifdim \wd\@tempboxa >\hsize% +% if caption is longer than a line, let it wrap around +\setbox\@tempboxa\hbox{\normalfont\sffamily\normalsize {#1.}~ }% +\parbox[t]{\hsize}{\normalfont\sffamily\normalsize \noindent\unhbox\@tempboxa#2}% +% if caption is shorter than a line, left justify +\else% +\hbox to\hsize{\normalfont\sffamily\normalsize\box\@tempboxa\hfil}% +\fi\fi} +\fi + +\else% traditional noncompsoc \@makecaption +\long\def\@makecaption#1#2{% +% test if is a for a figure or table +\ifx\@captype\@IEEEtablestring% +% if a table, do table caption +\footnotesize{\centering\normalfont\footnotesize#1.\qquad\scshape #2\par}% +\@IEEEtablecaptionsepspace +% if not a table, format it as a figure +\else +\@IEEEfigurecaptionsepspace +% 3/2001 use footnotesize, not small; use two nonbreaking spaces, not one +\setbox\@tempboxa\hbox{\normalfont\footnotesize {#1.}~~ #2}% +\ifdim \wd\@tempboxa >\hsize% +% if caption is longer than a line, let it wrap around +\setbox\@tempboxa\hbox{\normalfont\footnotesize {#1.}~~ }% +\parbox[t]{\hsize}{\normalfont\footnotesize\noindent\unhbox\@tempboxa#2}% +% if caption is shorter than a line, center if conference, left justify otherwise +\else% +\ifCLASSOPTIONconference \hbox to\hsize{\normalfont\footnotesize\box\@tempboxa\hfil}% +\else \hbox to\hsize{\normalfont\footnotesize\box\@tempboxa\hfil}% +\fi\fi\fi} +\fi + + + +% V1.7 disable captions class option, do so in a way that retains operation of \label +% within \caption +\ifCLASSOPTIONcaptionsoff +\long\def\@makecaption#1#2{\vspace*{2em}\footnotesize\begin{center}{\footnotesize #1}\end{center}% +\let\@IEEEtemporiglabeldefsave\label +\let\@IEEEtemplabelargsave\relax +\def\label##1{\gdef\@IEEEtemplabelargsave{##1}}% +\setbox\@tempboxa\hbox{#2}% +\let\label\@IEEEtemporiglabeldefsave +\ifx\@IEEEtemplabelargsave\relax\else\label{\@IEEEtemplabelargsave}\fi} +\fi + + +% V1.7 define end environments with \def not \let so as to work OK with +% preview-latex +\newcounter{figure} +\def\thefigure{\@arabic\c@figure} +\def\fps@figure{tbp} +\def\ftype@figure{1} +\def\ext@figure{lof} +\def\fnum@figure{\figurename~\thefigure} +\def\figure{\@float{figure}} +\def\endfigure{\end@float} +\@namedef{figure*}{\@dblfloat{figure}} +\@namedef{endfigure*}{\end@dblfloat} +\newcounter{table} +\ifCLASSOPTIONcompsoc +\def\thetable{\arabic{table}} +\else +\def\thetable{\@Roman\c@table} +\fi +\def\fps@table{tbp} +\def\ftype@table{2} +\def\ext@table{lot} +\def\fnum@table{\tablename~\thetable} +% V1.6 IEEE uses 8pt text for tables +% to default to footnotesize, we hack into LaTeX2e's \@floatboxreset and pray +\def\table{\def\@floatboxreset{\reset@font\scriptsize\@setminipage}% + \let\@makefntext\@maketablefntext + \@float{table}} +\def\endtable{\end@float} +% v1.6b double column tables need to default to footnotesize as well. +\@namedef{table*}{\def\@floatboxreset{\reset@font\scriptsize\@setminipage}\@dblfloat{table}} +\@namedef{endtable*}{\end@dblfloat} + + + + +%% +%% START OF IEEEeqnarry DEFINITIONS +%% +%% Inspired by the concepts, examples, and previous works of LaTeX +%% coders and developers such as Donald Arseneau, Fred Bartlett, +%% David Carlisle, Tony Liu, Frank Mittelbach, Piet van Oostrum, +%% Roland Winkler and Mark Wooding. +%% I don't make the claim that my work here is even near their calibre. ;) + + +% hook to allow easy changeover to IEEEtran.cls/tools.sty error reporting +\def\@IEEEclspkgerror{\ClassError{IEEEtran}} + +\newif\if@IEEEeqnarraystarform% flag to indicate if the environment was called as the star form +\@IEEEeqnarraystarformfalse + +\newif\if@advanceIEEEeqncolcnt% tracks if the environment should advance the col counter +% allows a way to make an \IEEEeqnarraybox that can be used within an \IEEEeqnarray +% used by IEEEeqnarraymulticol so that it can work properly in both +\@advanceIEEEeqncolcnttrue + +\newcount\@IEEEeqnnumcols % tracks how many IEEEeqnarray cols are defined +\newcount\@IEEEeqncolcnt % tracks how many IEEEeqnarray cols the user actually used + + +% The default math style used by the columns +\def\IEEEeqnarraymathstyle{\displaystyle} +% The default text style used by the columns +% default to using the current font +\def\IEEEeqnarraytextstyle{\relax} + +% like the iedlistdecl but for \IEEEeqnarray +\def\IEEEeqnarraydecl{\relax} +\def\IEEEeqnarrayboxdecl{\relax} + +% \yesnumber is the opposite of \nonumber +% a novel concept with the same def as the equationarray package +% However, we give IEEE versions too since some LaTeX packages such as +% the MDWtools mathenv.sty redefine \nonumber to something else. +\providecommand{\yesnumber}{\global\@eqnswtrue} +\def\IEEEyesnumber{\global\@eqnswtrue} +\def\IEEEnonumber{\global\@eqnswfalse} + + +\def\IEEEyessubnumber{\global\@IEEEissubequationtrue\global\@eqnswtrue% +\if@IEEEeqnarrayISinner% only do something inside an IEEEeqnarray +\if@IEEElastlinewassubequation\addtocounter{equation}{-1}\else\setcounter{IEEEsubequation}{1}\fi% +\def\@currentlabel{\p@IEEEsubequation\theIEEEsubequation}\fi} + +% flag to indicate that an equation is a sub equation +\newif\if@IEEEissubequation% +\@IEEEissubequationfalse + +% allows users to "push away" equations that get too close to the equation numbers +\def\IEEEeqnarraynumspace{\hphantom{\if@IEEEissubequation\theIEEEsubequationdis\else\theequationdis\fi}} + +% provides a way to span multiple columns within IEEEeqnarray environments +% will consider \if@advanceIEEEeqncolcnt before globally advancing the +% column counter - so as to work within \IEEEeqnarraybox +% usage: \IEEEeqnarraymulticol{number cols. to span}{col type}{cell text} +\long\def\IEEEeqnarraymulticol#1#2#3{\multispan{#1}% +% check if column is defined +\relax\expandafter\ifx\csname @IEEEeqnarraycolDEF#2\endcsname\@IEEEeqnarraycolisdefined% +\csname @IEEEeqnarraycolPRE#2\endcsname#3\relax\relax\relax\relax\relax% +\relax\relax\relax\relax\relax\csname @IEEEeqnarraycolPOST#2\endcsname% +\else% if not, error and use default type +\@IEEEclspkgerror{Invalid column type "#2" in \string\IEEEeqnarraymulticol.\MessageBreak +Using a default centering column instead}% +{You must define IEEEeqnarray column types before use.}% +\csname @IEEEeqnarraycolPRE@IEEEdefault\endcsname#3\relax\relax\relax\relax\relax% +\relax\relax\relax\relax\relax\csname @IEEEeqnarraycolPOST@IEEEdefault\endcsname% +\fi% +% advance column counter only if the IEEEeqnarray environment wants it +\if@advanceIEEEeqncolcnt\global\advance\@IEEEeqncolcnt by #1\relax\fi} + +% like \omit, but maintains track of the column counter for \IEEEeqnarray +\def\IEEEeqnarrayomit{\omit\if@advanceIEEEeqncolcnt\global\advance\@IEEEeqncolcnt by 1\relax\fi} + + +% provides a way to define a letter referenced column type +% usage: \IEEEeqnarraydefcol{col. type letter/name}{pre insertion text}{post insertion text} +\def\IEEEeqnarraydefcol#1#2#3{\expandafter\def\csname @IEEEeqnarraycolPRE#1\endcsname{#2}% +\expandafter\def\csname @IEEEeqnarraycolPOST#1\endcsname{#3}% +\expandafter\def\csname @IEEEeqnarraycolDEF#1\endcsname{1}} + + +% provides a way to define a numerically referenced inter-column glue types +% usage: \IEEEeqnarraydefcolsep{col. glue number}{glue definition} +\def\IEEEeqnarraydefcolsep#1#2{\expandafter\def\csname @IEEEeqnarraycolSEP\romannumeral #1\endcsname{#2}% +\expandafter\def\csname @IEEEeqnarraycolSEPDEF\romannumeral #1\endcsname{1}} + + +\def\@IEEEeqnarraycolisdefined{1}% just a macro for 1, used for checking undefined column types + + +% expands and appends the given argument to the \@IEEEtrantmptoksA token list +% used to build up the \halign preamble +\def\@IEEEappendtoksA#1{\edef\@@IEEEappendtoksA{\@IEEEtrantmptoksA={\the\@IEEEtrantmptoksA #1}}% +\@@IEEEappendtoksA} + +% also appends to \@IEEEtrantmptoksA, but does not expand the argument +% uses \toks8 as a scratchpad register +\def\@IEEEappendNOEXPANDtoksA#1{\toks8={#1}% +\edef\@@IEEEappendNOEXPANDtoksA{\@IEEEtrantmptoksA={\the\@IEEEtrantmptoksA\the\toks8}}% +\@@IEEEappendNOEXPANDtoksA} + +% define some common column types for the user +% math +\IEEEeqnarraydefcol{l}{$\IEEEeqnarraymathstyle}{$\hfil} +\IEEEeqnarraydefcol{c}{\hfil$\IEEEeqnarraymathstyle}{$\hfil} +\IEEEeqnarraydefcol{r}{\hfil$\IEEEeqnarraymathstyle}{$} +\IEEEeqnarraydefcol{L}{$\IEEEeqnarraymathstyle{}}{{}$\hfil} +\IEEEeqnarraydefcol{C}{\hfil$\IEEEeqnarraymathstyle{}}{{}$\hfil} +\IEEEeqnarraydefcol{R}{\hfil$\IEEEeqnarraymathstyle{}}{{}$} +% text +\IEEEeqnarraydefcol{s}{\IEEEeqnarraytextstyle}{\hfil} +\IEEEeqnarraydefcol{t}{\hfil\IEEEeqnarraytextstyle}{\hfil} +\IEEEeqnarraydefcol{u}{\hfil\IEEEeqnarraytextstyle}{} + +% vertical rules +\IEEEeqnarraydefcol{v}{}{\vrule width\arrayrulewidth} +\IEEEeqnarraydefcol{vv}{\vrule width\arrayrulewidth\hfil}{\hfil\vrule width\arrayrulewidth} +\IEEEeqnarraydefcol{V}{}{\vrule width\arrayrulewidth\hskip\doublerulesep\vrule width\arrayrulewidth} +\IEEEeqnarraydefcol{VV}{\vrule width\arrayrulewidth\hskip\doublerulesep\vrule width\arrayrulewidth\hfil}% +{\hfil\vrule width\arrayrulewidth\hskip\doublerulesep\vrule width\arrayrulewidth} + +% horizontal rules +\IEEEeqnarraydefcol{h}{}{\leaders\hrule height\arrayrulewidth\hfil} +\IEEEeqnarraydefcol{H}{}{\leaders\vbox{\hrule width\arrayrulewidth\vskip\doublerulesep\hrule width\arrayrulewidth}\hfil} + +% plain +\IEEEeqnarraydefcol{x}{}{} +\IEEEeqnarraydefcol{X}{$}{$} + +% the default column type to use in the event a column type is not defined +\IEEEeqnarraydefcol{@IEEEdefault}{\hfil$\IEEEeqnarraymathstyle}{$\hfil} + + +% a zero tabskip (used for "-" col types) +\def\@IEEEeqnarraycolSEPzero{0pt plus 0pt minus 0pt} +% a centering tabskip (used for "+" col types) +\def\@IEEEeqnarraycolSEPcenter{1000pt plus 0pt minus 1000pt} + +% top level default tabskip glues for the start, end, and inter-column +% may be reset within environments not always at the top level, e.g., \IEEEeqnarraybox +\edef\@IEEEeqnarraycolSEPdefaultstart{\@IEEEeqnarraycolSEPcenter}% default start glue +\edef\@IEEEeqnarraycolSEPdefaultend{\@IEEEeqnarraycolSEPcenter}% default end glue +\edef\@IEEEeqnarraycolSEPdefaultmid{\@IEEEeqnarraycolSEPzero}% default inter-column glue + + + +% creates a vertical rule that extends from the bottom to the top a a cell +% Provided in case other packages redefine \vline some other way. +% usage: \IEEEeqnarrayvrule[rule thickness] +% If no argument is provided, \arrayrulewidth will be used for the rule thickness. +\newcommand\IEEEeqnarrayvrule[1][\arrayrulewidth]{\vrule\@width#1\relax} + +% creates a blank separator row +% usage: \IEEEeqnarrayseprow[separation length][font size commands] +% default is \IEEEeqnarrayseprow[0.25\normalbaselineskip][\relax] +% blank arguments inherit the default values +% uses \skip5 as a scratch register - calls \@IEEEeqnarraystrutsize which uses more scratch registers +\def\IEEEeqnarrayseprow{\relax\@ifnextchar[{\@IEEEeqnarrayseprow}{\@IEEEeqnarrayseprow[0.25\normalbaselineskip]}} +\def\@IEEEeqnarrayseprow[#1]{\relax\@ifnextchar[{\@@IEEEeqnarrayseprow[#1]}{\@@IEEEeqnarrayseprow[#1][\relax]}} +\def\@@IEEEeqnarrayseprow[#1][#2]{\def\@IEEEeqnarrayseprowARGONE{#1}% +\ifx\@IEEEeqnarrayseprowARGONE\@empty% +% get the skip value, based on the font commands +% use skip5 because \IEEEeqnarraystrutsize uses \skip0, \skip2, \skip3 +% assign within a bogus box to confine the font changes +{\setbox0=\hbox{#2\relax\global\skip5=0.25\normalbaselineskip}}% +\else% +{\setbox0=\hbox{#2\relax\global\skip5=#1}}% +\fi% +\@IEEEeqnarrayhoptolastcolumn\IEEEeqnarraystrutsize{\skip5}{0pt}[\relax]\relax} + +% creates a blank separator row, but omits all the column templates +% usage: \IEEEeqnarrayseprowcut[separation length][font size commands] +% default is \IEEEeqnarrayseprowcut[0.25\normalbaselineskip][\relax] +% blank arguments inherit the default values +% uses \skip5 as a scratch register - calls \@IEEEeqnarraystrutsize which uses more scratch registers +\def\IEEEeqnarrayseprowcut{\multispan{\@IEEEeqnnumcols}\relax% span all the cols +% advance column counter only if the IEEEeqnarray environment wants it +\if@advanceIEEEeqncolcnt\global\advance\@IEEEeqncolcnt by \@IEEEeqnnumcols\relax\fi% +\@ifnextchar[{\@IEEEeqnarrayseprowcut}{\@IEEEeqnarrayseprowcut[0.25\normalbaselineskip]}} +\def\@IEEEeqnarrayseprowcut[#1]{\relax\@ifnextchar[{\@@IEEEeqnarrayseprowcut[#1]}{\@@IEEEeqnarrayseprowcut[#1][\relax]}} +\def\@@IEEEeqnarrayseprowcut[#1][#2]{\def\@IEEEeqnarrayseprowARGONE{#1}% +\ifx\@IEEEeqnarrayseprowARGONE\@empty% +% get the skip value, based on the font commands +% use skip5 because \IEEEeqnarraystrutsize uses \skip0, \skip2, \skip3 +% assign within a bogus box to confine the font changes +{\setbox0=\hbox{#2\relax\global\skip5=0.25\normalbaselineskip}}% +\else% +{\setbox0=\hbox{#2\relax\global\skip5=#1}}% +\fi% +\IEEEeqnarraystrutsize{\skip5}{0pt}[\relax]\relax} + + + +% draws a single rule across all the columns optional +% argument determines the rule width, \arrayrulewidth is the default +% updates column counter as needed and turns off struts +% usage: \IEEEeqnarrayrulerow[rule line thickness] +\def\IEEEeqnarrayrulerow{\multispan{\@IEEEeqnnumcols}\relax% span all the cols +% advance column counter only if the IEEEeqnarray environment wants it +\if@advanceIEEEeqncolcnt\global\advance\@IEEEeqncolcnt by \@IEEEeqnnumcols\relax\fi% +\@ifnextchar[{\@IEEEeqnarrayrulerow}{\@IEEEeqnarrayrulerow[\arrayrulewidth]}} +\def\@IEEEeqnarrayrulerow[#1]{\leaders\hrule height#1\hfil\relax% put in our rule +% turn off any struts +\IEEEeqnarraystrutsize{0pt}{0pt}[\relax]\relax} + + +% draws a double rule by using a single rule row, a separator row, and then +% another single rule row +% first optional argument determines the rule thicknesses, \arrayrulewidth is the default +% second optional argument determines the rule spacing, \doublerulesep is the default +% usage: \IEEEeqnarraydblrulerow[rule line thickness][rule spacing] +\def\IEEEeqnarraydblrulerow{\multispan{\@IEEEeqnnumcols}\relax% span all the cols +% advance column counter only if the IEEEeqnarray environment wants it +\if@advanceIEEEeqncolcnt\global\advance\@IEEEeqncolcnt by \@IEEEeqnnumcols\relax\fi% +\@ifnextchar[{\@IEEEeqnarraydblrulerow}{\@IEEEeqnarraydblrulerow[\arrayrulewidth]}} +\def\@IEEEeqnarraydblrulerow[#1]{\relax\@ifnextchar[{\@@IEEEeqnarraydblrulerow[#1]}% +{\@@IEEEeqnarraydblrulerow[#1][\doublerulesep]}} +\def\@@IEEEeqnarraydblrulerow[#1][#2]{\def\@IEEEeqnarraydblrulerowARG{#1}% +% we allow the user to say \IEEEeqnarraydblrulerow[][] +\ifx\@IEEEeqnarraydblrulerowARG\@empty% +\@IEEEeqnarrayrulerow[\arrayrulewidth]% +\else% +\@IEEEeqnarrayrulerow[#1]\relax% +\fi% +\def\@IEEEeqnarraydblrulerowARG{#2}% +\ifx\@IEEEeqnarraydblrulerowARG\@empty% +\\\IEEEeqnarrayseprow[\doublerulesep][\relax]% +\else% +\\\IEEEeqnarrayseprow[#2][\relax]% +\fi% +\\\multispan{\@IEEEeqnnumcols}% +% advance column counter only if the IEEEeqnarray environment wants it +\if@advanceIEEEeqncolcnt\global\advance\@IEEEeqncolcnt by \@IEEEeqnnumcols\relax\fi% +\def\@IEEEeqnarraydblrulerowARG{#1}% +\ifx\@IEEEeqnarraydblrulerowARG\@empty% +\@IEEEeqnarrayrulerow[\arrayrulewidth]% +\else% +\@IEEEeqnarrayrulerow[#1]% +\fi% +} + +% draws a double rule by using a single rule row, a separator (cutting) row, and then +% another single rule row +% first optional argument determines the rule thicknesses, \arrayrulewidth is the default +% second optional argument determines the rule spacing, \doublerulesep is the default +% usage: \IEEEeqnarraydblrulerow[rule line thickness][rule spacing] +\def\IEEEeqnarraydblrulerowcut{\multispan{\@IEEEeqnnumcols}\relax% span all the cols +% advance column counter only if the IEEEeqnarray environment wants it +\if@advanceIEEEeqncolcnt\global\advance\@IEEEeqncolcnt by \@IEEEeqnnumcols\relax\fi% +\@ifnextchar[{\@IEEEeqnarraydblrulerowcut}{\@IEEEeqnarraydblrulerowcut[\arrayrulewidth]}} +\def\@IEEEeqnarraydblrulerowcut[#1]{\relax\@ifnextchar[{\@@IEEEeqnarraydblrulerowcut[#1]}% +{\@@IEEEeqnarraydblrulerowcut[#1][\doublerulesep]}} +\def\@@IEEEeqnarraydblrulerowcut[#1][#2]{\def\@IEEEeqnarraydblrulerowARG{#1}% +% we allow the user to say \IEEEeqnarraydblrulerow[][] +\ifx\@IEEEeqnarraydblrulerowARG\@empty% +\@IEEEeqnarrayrulerow[\arrayrulewidth]% +\else% +\@IEEEeqnarrayrulerow[#1]% +\fi% +\def\@IEEEeqnarraydblrulerowARG{#2}% +\ifx\@IEEEeqnarraydblrulerowARG\@empty% +\\\IEEEeqnarrayseprowcut[\doublerulesep][\relax]% +\else% +\\\IEEEeqnarrayseprowcut[#2][\relax]% +\fi% +\\\multispan{\@IEEEeqnnumcols}% +% advance column counter only if the IEEEeqnarray environment wants it +\if@advanceIEEEeqncolcnt\global\advance\@IEEEeqncolcnt by \@IEEEeqnnumcols\relax\fi% +\def\@IEEEeqnarraydblrulerowARG{#1}% +\ifx\@IEEEeqnarraydblrulerowARG\@empty% +\@IEEEeqnarrayrulerow[\arrayrulewidth]% +\else% +\@IEEEeqnarrayrulerow[#1]% +\fi% +} + + + +% inserts a full row's worth of &'s +% relies on \@IEEEeqnnumcols to provide the correct number of columns +% uses \@IEEEtrantmptoksA, \count0 as scratch registers +\def\@IEEEeqnarrayhoptolastcolumn{\@IEEEtrantmptoksA={}\count0=1\relax% +\loop% add cols if the user did not use them all +\ifnum\count0<\@IEEEeqnnumcols\relax% +\@IEEEappendtoksA{&}% +\advance\count0 by 1\relax% update the col count +\repeat% +\the\@IEEEtrantmptoksA%execute the &'s +} + + + +\newif\if@IEEEeqnarrayISinner % flag to indicate if we are within the lines +\@IEEEeqnarrayISinnerfalse % of an IEEEeqnarray - after the IEEEeqnarraydecl + +\edef\@IEEEeqnarrayTHEstrutheight{0pt} % height and depth of IEEEeqnarray struts +\edef\@IEEEeqnarrayTHEstrutdepth{0pt} + +\edef\@IEEEeqnarrayTHEmasterstrutheight{0pt} % default height and depth of +\edef\@IEEEeqnarrayTHEmasterstrutdepth{0pt} % struts within an IEEEeqnarray + +\edef\@IEEEeqnarrayTHEmasterstrutHSAVE{0pt} % saved master strut height +\edef\@IEEEeqnarrayTHEmasterstrutDSAVE{0pt} % and depth + +\newif\if@IEEEeqnarrayusemasterstrut % flag to indicate that the master strut value +\@IEEEeqnarrayusemasterstruttrue % is to be used + + + +% saves the strut height and depth of the master strut +\def\@IEEEeqnarraymasterstrutsave{\relax% +\expandafter\skip0=\@IEEEeqnarrayTHEmasterstrutheight\relax% +\expandafter\skip2=\@IEEEeqnarrayTHEmasterstrutdepth\relax% +% remove stretchability +\dimen0\skip0\relax% +\dimen2\skip2\relax% +% save values +\edef\@IEEEeqnarrayTHEmasterstrutHSAVE{\the\dimen0}% +\edef\@IEEEeqnarrayTHEmasterstrutDSAVE{\the\dimen2}} + +% restores the strut height and depth of the master strut +\def\@IEEEeqnarraymasterstrutrestore{\relax% +\expandafter\skip0=\@IEEEeqnarrayTHEmasterstrutHSAVE\relax% +\expandafter\skip2=\@IEEEeqnarrayTHEmasterstrutDSAVE\relax% +% remove stretchability +\dimen0\skip0\relax% +\dimen2\skip2\relax% +% restore values +\edef\@IEEEeqnarrayTHEmasterstrutheight{\the\dimen0}% +\edef\@IEEEeqnarrayTHEmasterstrutdepth{\the\dimen2}} + + +% globally restores the strut height and depth to the +% master values and sets the master strut flag to true +\def\@IEEEeqnarraystrutreset{\relax% +\expandafter\skip0=\@IEEEeqnarrayTHEmasterstrutheight\relax% +\expandafter\skip2=\@IEEEeqnarrayTHEmasterstrutdepth\relax% +% remove stretchability +\dimen0\skip0\relax% +\dimen2\skip2\relax% +% restore values +\xdef\@IEEEeqnarrayTHEstrutheight{\the\dimen0}% +\xdef\@IEEEeqnarrayTHEstrutdepth{\the\dimen2}% +\global\@IEEEeqnarrayusemasterstruttrue} + + +% if the master strut is not to be used, make the current +% values of \@IEEEeqnarrayTHEstrutheight, \@IEEEeqnarrayTHEstrutdepth +% and the use master strut flag, global +% this allows user strut commands issued in the last column to be carried +% into the isolation/strut column +\def\@IEEEeqnarrayglobalizestrutstatus{\relax% +\if@IEEEeqnarrayusemasterstrut\else% +\xdef\@IEEEeqnarrayTHEstrutheight{\@IEEEeqnarrayTHEstrutheight}% +\xdef\@IEEEeqnarrayTHEstrutdepth{\@IEEEeqnarrayTHEstrutdepth}% +\global\@IEEEeqnarrayusemasterstrutfalse% +\fi} + + + +% usage: \IEEEeqnarraystrutsize{height}{depth}[font size commands] +% If called outside the lines of an IEEEeqnarray, sets the height +% and depth of both the master and local struts. If called inside +% an IEEEeqnarray line, sets the height and depth of the local strut +% only and sets the flag to indicate the use of the local strut +% values. If the height or depth is left blank, 0.7\normalbaselineskip +% and 0.3\normalbaselineskip will be used, respectively. +% The optional argument can be used to evaluate the lengths under +% a different font size and styles. If none is specified, the current +% font is used. +% uses scratch registers \skip0, \skip2, \skip3, \dimen0, \dimen2 +\def\IEEEeqnarraystrutsize#1#2{\relax\@ifnextchar[{\@IEEEeqnarraystrutsize{#1}{#2}}{\@IEEEeqnarraystrutsize{#1}{#2}[\relax]}} +\def\@IEEEeqnarraystrutsize#1#2[#3]{\def\@IEEEeqnarraystrutsizeARG{#1}% +\ifx\@IEEEeqnarraystrutsizeARG\@empty% +{\setbox0=\hbox{#3\relax\global\skip3=0.7\normalbaselineskip}}% +\skip0=\skip3\relax% +\else% arg one present +{\setbox0=\hbox{#3\relax\global\skip3=#1\relax}}% +\skip0=\skip3\relax% +\fi% if null arg +\def\@IEEEeqnarraystrutsizeARG{#2}% +\ifx\@IEEEeqnarraystrutsizeARG\@empty% +{\setbox0=\hbox{#3\relax\global\skip3=0.3\normalbaselineskip}}% +\skip2=\skip3\relax% +\else% arg two present +{\setbox0=\hbox{#3\relax\global\skip3=#2\relax}}% +\skip2=\skip3\relax% +\fi% if null arg +% remove stretchability, just to be safe +\dimen0\skip0\relax% +\dimen2\skip2\relax% +% dimen0 = height, dimen2 = depth +\if@IEEEeqnarrayISinner% inner does not touch master strut size +\edef\@IEEEeqnarrayTHEstrutheight{\the\dimen0}% +\edef\@IEEEeqnarrayTHEstrutdepth{\the\dimen2}% +\@IEEEeqnarrayusemasterstrutfalse% do not use master +\else% outer, have to set master strut too +\edef\@IEEEeqnarrayTHEmasterstrutheight{\the\dimen0}% +\edef\@IEEEeqnarrayTHEmasterstrutdepth{\the\dimen2}% +\edef\@IEEEeqnarrayTHEstrutheight{\the\dimen0}% +\edef\@IEEEeqnarrayTHEstrutdepth{\the\dimen2}% +\@IEEEeqnarrayusemasterstruttrue% use master strut +\fi} + + +% usage: \IEEEeqnarraystrutsizeadd{added height}{added depth}[font size commands] +% If called outside the lines of an IEEEeqnarray, adds the given height +% and depth to both the master and local struts. +% If called inside an IEEEeqnarray line, adds the given height and depth +% to the local strut only and sets the flag to indicate the use +% of the local strut values. +% In both cases, if a height or depth is left blank, 0pt is used instead. +% The optional argument can be used to evaluate the lengths under +% a different font size and styles. If none is specified, the current +% font is used. +% uses scratch registers \skip0, \skip2, \skip3, \dimen0, \dimen2 +\def\IEEEeqnarraystrutsizeadd#1#2{\relax\@ifnextchar[{\@IEEEeqnarraystrutsizeadd{#1}{#2}}{\@IEEEeqnarraystrutsizeadd{#1}{#2}[\relax]}} +\def\@IEEEeqnarraystrutsizeadd#1#2[#3]{\def\@IEEEeqnarraystrutsizearg{#1}% +\ifx\@IEEEeqnarraystrutsizearg\@empty% +\skip0=0pt\relax% +\else% arg one present +{\setbox0=\hbox{#3\relax\global\skip3=#1}}% +\skip0=\skip3\relax% +\fi% if null arg +\def\@IEEEeqnarraystrutsizearg{#2}% +\ifx\@IEEEeqnarraystrutsizearg\@empty% +\skip2=0pt\relax% +\else% arg two present +{\setbox0=\hbox{#3\relax\global\skip3=#2}}% +\skip2=\skip3\relax% +\fi% if null arg +% remove stretchability, just to be safe +\dimen0\skip0\relax% +\dimen2\skip2\relax% +% dimen0 = height, dimen2 = depth +\if@IEEEeqnarrayISinner% inner does not touch master strut size +% get local strut size +\expandafter\skip0=\@IEEEeqnarrayTHEstrutheight\relax% +\expandafter\skip2=\@IEEEeqnarrayTHEstrutdepth\relax% +% add it to the user supplied values +\advance\dimen0 by \skip0\relax% +\advance\dimen2 by \skip2\relax% +% update the local strut size +\edef\@IEEEeqnarrayTHEstrutheight{\the\dimen0}% +\edef\@IEEEeqnarrayTHEstrutdepth{\the\dimen2}% +\@IEEEeqnarrayusemasterstrutfalse% do not use master +\else% outer, have to set master strut too +% get master strut size +\expandafter\skip0=\@IEEEeqnarrayTHEmasterstrutheight\relax% +\expandafter\skip2=\@IEEEeqnarrayTHEmasterstrutdepth\relax% +% add it to the user supplied values +\advance\dimen0 by \skip0\relax% +\advance\dimen2 by \skip2\relax% +% update the local and master strut sizes +\edef\@IEEEeqnarrayTHEmasterstrutheight{\the\dimen0}% +\edef\@IEEEeqnarrayTHEmasterstrutdepth{\the\dimen2}% +\edef\@IEEEeqnarrayTHEstrutheight{\the\dimen0}% +\edef\@IEEEeqnarrayTHEstrutdepth{\the\dimen2}% +\@IEEEeqnarrayusemasterstruttrue% use master strut +\fi} + + +% allow user a way to see the struts +\newif\ifIEEEvisiblestruts +\IEEEvisiblestrutsfalse + +% inserts an invisible strut using the master or local strut values +% uses scratch registers \skip0, \skip2, \dimen0, \dimen2 +\def\@IEEEeqnarrayinsertstrut{\relax% +\if@IEEEeqnarrayusemasterstrut +% get master strut size +\expandafter\skip0=\@IEEEeqnarrayTHEmasterstrutheight\relax% +\expandafter\skip2=\@IEEEeqnarrayTHEmasterstrutdepth\relax% +\else% +% get local strut size +\expandafter\skip0=\@IEEEeqnarrayTHEstrutheight\relax% +\expandafter\skip2=\@IEEEeqnarrayTHEstrutdepth\relax% +\fi% +% remove stretchability, probably not needed +\dimen0\skip0\relax% +\dimen2\skip2\relax% +% dimen0 = height, dimen2 = depth +% allow user to see struts if desired +\ifIEEEvisiblestruts% +\vrule width0.2pt height\dimen0 depth\dimen2\relax% +\else% +\vrule width0pt height\dimen0 depth\dimen2\relax\fi} + + +% creates an invisible strut, useable even outside \IEEEeqnarray +% if \IEEEvisiblestrutstrue, the strut will be visible and 0.2pt wide. +% usage: \IEEEstrut[height][depth][font size commands] +% default is \IEEEstrut[0.7\normalbaselineskip][0.3\normalbaselineskip][\relax] +% blank arguments inherit the default values +% uses \dimen0, \dimen2, \skip0, \skip2 +\def\IEEEstrut{\relax\@ifnextchar[{\@IEEEstrut}{\@IEEEstrut[0.7\normalbaselineskip]}} +\def\@IEEEstrut[#1]{\relax\@ifnextchar[{\@@IEEEstrut[#1]}{\@@IEEEstrut[#1][0.3\normalbaselineskip]}} +\def\@@IEEEstrut[#1][#2]{\relax\@ifnextchar[{\@@@IEEEstrut[#1][#2]}{\@@@IEEEstrut[#1][#2][\relax]}} +\def\@@@IEEEstrut[#1][#2][#3]{\mbox{#3\relax% +\def\@IEEEstrutARG{#1}% +\ifx\@IEEEstrutARG\@empty% +\skip0=0.7\normalbaselineskip\relax% +\else% +\skip0=#1\relax% +\fi% +\def\@IEEEstrutARG{#2}% +\ifx\@IEEEstrutARG\@empty% +\skip2=0.3\normalbaselineskip\relax% +\else% +\skip2=#2\relax% +\fi% +% remove stretchability, probably not needed +\dimen0\skip0\relax% +\dimen2\skip2\relax% +\ifIEEEvisiblestruts% +\vrule width0.2pt height\dimen0 depth\dimen2\relax% +\else% +\vrule width0.0pt height\dimen0 depth\dimen2\relax\fi}} + + +% enables strut mode by setting a default strut size and then zeroing the +% \baselineskip, \lineskip, \lineskiplimit and \jot +\def\IEEEeqnarraystrutmode{\IEEEeqnarraystrutsize{0.7\normalbaselineskip}{0.3\normalbaselineskip}[\relax]% +\baselineskip=0pt\lineskip=0pt\lineskiplimit=0pt\jot=0pt} + + + +\def\IEEEeqnarray{\@IEEEeqnarraystarformfalse\@IEEEeqnarray} +\def\endIEEEeqnarray{\end@IEEEeqnarray} + +\@namedef{IEEEeqnarray*}{\@IEEEeqnarraystarformtrue\@IEEEeqnarray} +\@namedef{endIEEEeqnarray*}{\end@IEEEeqnarray} + + +% \IEEEeqnarray is an enhanced \eqnarray. +% The star form defaults to not putting equation numbers at the end of each row. +% usage: \IEEEeqnarray[decl]{cols} +\def\@IEEEeqnarray{\relax\@ifnextchar[{\@@IEEEeqnarray}{\@@IEEEeqnarray[\relax]}} +\def\@@IEEEeqnarray[#1]#2{% + % default to showing the equation number or not based on whether or not + % the star form was involked + \if@IEEEeqnarraystarform\global\@eqnswfalse + \else% not the star form + \global\@eqnswtrue + \fi% if star form + \@IEEEissubequationfalse% default to no subequations + \@IEEElastlinewassubequationfalse% assume last line is not a sub equation + \@IEEEeqnarrayISinnerfalse% not yet within the lines of the halign + \@IEEEeqnarraystrutsize{0pt}{0pt}[\relax]% turn off struts by default + \@IEEEeqnarrayusemasterstruttrue% use master strut till user asks otherwise + \IEEEvisiblestrutsfalse% diagnostic mode defaults to off + % no extra space unless the user specifically requests it + \lineskip=0pt\relax + \lineskiplimit=0pt\relax + \baselineskip=\normalbaselineskip\relax% + \jot=\IEEEnormaljot\relax% + \mathsurround\z@\relax% no extra spacing around math + \@advanceIEEEeqncolcnttrue% advance the col counter for each col the user uses, + % used in \IEEEeqnarraymulticol and in the preamble build + \stepcounter{equation}% advance equation counter before first line + \setcounter{IEEEsubequation}{0}% no subequation yet + \def\@currentlabel{\p@equation\theequation}% redefine the ref label + \IEEEeqnarraydecl\relax% allow a way for the user to make global overrides + #1\relax% allow user to override defaults + \let\\\@IEEEeqnarraycr% replace newline with one that can put in eqn. numbers + \global\@IEEEeqncolcnt\z@% col. count = 0 for first line + \@IEEEbuildpreamble #2\end\relax% build the preamble and put it into \@IEEEtrantmptoksA + % put in the column for the equation number + \ifnum\@IEEEeqnnumcols>0\relax\@IEEEappendtoksA{&}\fi% col separator for those after the first + \toks0={##}% + % advance the \@IEEEeqncolcnt for the isolation col, this helps with error checking + \@IEEEappendtoksA{\global\advance\@IEEEeqncolcnt by 1\relax}% + % add the isolation column + \@IEEEappendtoksA{\tabskip\z@skip\bgroup\the\toks0\egroup}% + % advance the \@IEEEeqncolcnt for the equation number col, this helps with error checking + \@IEEEappendtoksA{&\global\advance\@IEEEeqncolcnt by 1\relax}% + % add the equation number col to the preamble + \@IEEEappendtoksA{\tabskip\z@skip\hb@xt@\z@\bgroup\hss\the\toks0\egroup}% + % note \@IEEEeqnnumcols does not count the equation col or isolation col + % set the starting tabskip glue as determined by the preamble build + \tabskip=\@IEEEBPstartglue\relax + % begin the display alignment + \@IEEEeqnarrayISinnertrue% commands are now within the lines + $$\everycr{}\halign to\displaywidth\bgroup + % "exspand" the preamble + \span\the\@IEEEtrantmptoksA\cr} + +% enter isolation/strut column (or the next column if the user did not use +% every column), record the strut status, complete the columns, do the strut if needed, +% restore counters to correct values and exit +\def\end@IEEEeqnarray{\@IEEEeqnarrayglobalizestrutstatus&\@@IEEEeqnarraycr\egroup% +\if@IEEElastlinewassubequation\global\advance\c@IEEEsubequation\m@ne\fi% +\global\advance\c@equation\m@ne% +$$\@ignoretrue} + +% need a way to remember if last line is a subequation +\newif\if@IEEElastlinewassubequation% +\@IEEElastlinewassubequationfalse + +% IEEEeqnarray uses a modifed \\ instead of the plain \cr to +% end rows. This allows for things like \\*[vskip amount] +% This "cr" macros are modified versions those for LaTeX2e's eqnarray +% the {\ifnum0=`} braces must be kept away from the last column to avoid +% altering spacing of its math, so we use & to advance to the next column +% as there is an isolation/strut column after the user's columns +\def\@IEEEeqnarraycr{\@IEEEeqnarrayglobalizestrutstatus&% save strut status and advance to next column + {\ifnum0=`}\fi + \@ifstar{% + \global\@eqpen\@M\@IEEEeqnarrayYCR + }{% + \global\@eqpen\interdisplaylinepenalty \@IEEEeqnarrayYCR + }% +} + +\def\@IEEEeqnarrayYCR{\@testopt\@IEEEeqnarrayXCR\z@skip} + +\def\@IEEEeqnarrayXCR[#1]{% + \ifnum0=`{\fi}% + \@@IEEEeqnarraycr + \noalign{\penalty\@eqpen\vskip\jot\vskip #1\relax}}% + +\def\@@IEEEeqnarraycr{\@IEEEtrantmptoksA={}% clear token register + \advance\@IEEEeqncolcnt by -1\relax% adjust col count because of the isolation column + \ifnum\@IEEEeqncolcnt>\@IEEEeqnnumcols\relax + \@IEEEclspkgerror{Too many columns within the IEEEeqnarray\MessageBreak + environment}% + {Use fewer \string &'s or put more columns in the IEEEeqnarry column\MessageBreak + specifications.}\relax% + \else + \loop% add cols if the user did not use them all + \ifnum\@IEEEeqncolcnt<\@IEEEeqnnumcols\relax + \@IEEEappendtoksA{&}% + \advance\@IEEEeqncolcnt by 1\relax% update the col count + \repeat + % this number of &'s will take us the the isolation column + \fi + % execute the &'s + \the\@IEEEtrantmptoksA% + % handle the strut/isolation column + \@IEEEeqnarrayinsertstrut% do the strut if needed + \@IEEEeqnarraystrutreset% reset the strut system for next line or IEEEeqnarray + &% and enter the equation number column + % is this line needs an equation number, display it and advance the + % (sub)equation counters, record what type this line was + \if@eqnsw% + \if@IEEEissubequation\theIEEEsubequationdis\addtocounter{equation}{1}\stepcounter{IEEEsubequation}% + \global\@IEEElastlinewassubequationtrue% + \else% display a standard equation number, initialize the IEEEsubequation counter + \theequationdis\stepcounter{equation}\setcounter{IEEEsubequation}{0}% + \global\@IEEElastlinewassubequationfalse\fi% + \fi% + % reset the eqnsw flag to indicate default preference of the display of equation numbers + \if@IEEEeqnarraystarform\global\@eqnswfalse\else\global\@eqnswtrue\fi + \global\@IEEEissubequationfalse% reset the subequation flag + % reset the number of columns the user actually used + \global\@IEEEeqncolcnt\z@\relax + % the real end of the line + \cr} + + + + + +% \IEEEeqnarraybox is like \IEEEeqnarray except the box form puts everything +% inside a vtop, vbox, or vcenter box depending on the letter in the second +% optional argument (t,b,c). Vbox is the default. Unlike \IEEEeqnarray, +% equation numbers are not displayed and \IEEEeqnarraybox can be nested. +% \IEEEeqnarrayboxm is for math mode (like \array) and does not put the vbox +% within an hbox. +% \IEEEeqnarrayboxt is for text mode (like \tabular) and puts the vbox within +% a \hbox{$ $} construct. +% \IEEEeqnarraybox will auto detect whether to use \IEEEeqnarrayboxm or +% \IEEEeqnarrayboxt depending on the math mode. +% The third optional argument specifies the width this box is to be set to - +% natural width is the default. +% The * forms do not add \jot line spacing +% usage: \IEEEeqnarraybox[decl][pos][width]{cols} +\def\IEEEeqnarrayboxm{\@IEEEeqnarraystarformfalse\@IEEEeqnarrayboxHBOXSWfalse\@IEEEeqnarraybox} +\def\endIEEEeqnarrayboxm{\end@IEEEeqnarraybox} +\@namedef{IEEEeqnarrayboxm*}{\@IEEEeqnarraystarformtrue\@IEEEeqnarrayboxHBOXSWfalse\@IEEEeqnarraybox} +\@namedef{endIEEEeqnarrayboxm*}{\end@IEEEeqnarraybox} + +\def\IEEEeqnarrayboxt{\@IEEEeqnarraystarformfalse\@IEEEeqnarrayboxHBOXSWtrue\@IEEEeqnarraybox} +\def\endIEEEeqnarrayboxt{\end@IEEEeqnarraybox} +\@namedef{IEEEeqnarrayboxt*}{\@IEEEeqnarraystarformtrue\@IEEEeqnarrayboxHBOXSWtrue\@IEEEeqnarraybox} +\@namedef{endIEEEeqnarrayboxt*}{\end@IEEEeqnarraybox} + +\def\IEEEeqnarraybox{\@IEEEeqnarraystarformfalse\ifmmode\@IEEEeqnarrayboxHBOXSWfalse\else\@IEEEeqnarrayboxHBOXSWtrue\fi% +\@IEEEeqnarraybox} +\def\endIEEEeqnarraybox{\end@IEEEeqnarraybox} + +\@namedef{IEEEeqnarraybox*}{\@IEEEeqnarraystarformtrue\ifmmode\@IEEEeqnarrayboxHBOXSWfalse\else\@IEEEeqnarrayboxHBOXSWtrue\fi% +\@IEEEeqnarraybox} +\@namedef{endIEEEeqnarraybox*}{\end@IEEEeqnarraybox} + +% flag to indicate if the \IEEEeqnarraybox needs to put things into an hbox{$ $} +% for \vcenter in non-math mode +\newif\if@IEEEeqnarrayboxHBOXSW% +\@IEEEeqnarrayboxHBOXSWfalse + +\def\@IEEEeqnarraybox{\relax\@ifnextchar[{\@@IEEEeqnarraybox}{\@@IEEEeqnarraybox[\relax]}} +\def\@@IEEEeqnarraybox[#1]{\relax\@ifnextchar[{\@@@IEEEeqnarraybox[#1]}{\@@@IEEEeqnarraybox[#1][b]}} +\def\@@@IEEEeqnarraybox[#1][#2]{\relax\@ifnextchar[{\@@@@IEEEeqnarraybox[#1][#2]}{\@@@@IEEEeqnarraybox[#1][#2][\relax]}} + +% #1 = decl; #2 = t,b,c; #3 = width, #4 = col specs +\def\@@@@IEEEeqnarraybox[#1][#2][#3]#4{\@IEEEeqnarrayISinnerfalse % not yet within the lines of the halign + \@IEEEeqnarraymasterstrutsave% save current master strut values + \@IEEEeqnarraystrutsize{0pt}{0pt}[\relax]% turn off struts by default + \@IEEEeqnarrayusemasterstruttrue% use master strut till user asks otherwise + \IEEEvisiblestrutsfalse% diagnostic mode defaults to off + % no extra space unless the user specifically requests it + \lineskip=0pt\relax% + \lineskiplimit=0pt\relax% + \baselineskip=\normalbaselineskip\relax% + \jot=\IEEEnormaljot\relax% + \mathsurround\z@\relax% no extra spacing around math + % the default end glues are zero for an \IEEEeqnarraybox + \edef\@IEEEeqnarraycolSEPdefaultstart{\@IEEEeqnarraycolSEPzero}% default start glue + \edef\@IEEEeqnarraycolSEPdefaultend{\@IEEEeqnarraycolSEPzero}% default end glue + \edef\@IEEEeqnarraycolSEPdefaultmid{\@IEEEeqnarraycolSEPzero}% default inter-column glue + \@advanceIEEEeqncolcntfalse% do not advance the col counter for each col the user uses, + % used in \IEEEeqnarraymulticol and in the preamble build + \IEEEeqnarrayboxdecl\relax% allow a way for the user to make global overrides + #1\relax% allow user to override defaults + \let\\\@IEEEeqnarrayboxcr% replace newline with one that allows optional spacing + \@IEEEbuildpreamble #4\end\relax% build the preamble and put it into \@IEEEtrantmptoksA + % add an isolation column to the preamble to stop \\'s {} from getting into the last col + \ifnum\@IEEEeqnnumcols>0\relax\@IEEEappendtoksA{&}\fi% col separator for those after the first + \toks0={##}% + % add the isolation column to the preamble + \@IEEEappendtoksA{\tabskip\z@skip\bgroup\the\toks0\egroup}% + % set the starting tabskip glue as determined by the preamble build + \tabskip=\@IEEEBPstartglue\relax + % begin the alignment + \everycr{}% + % use only the very first token to determine the positioning + % this stops some problems when the user uses more than one letter, + % but is probably not worth the effort + % \noindent is used as a delimiter + \def\@IEEEgrabfirstoken##1##2\noindent{\let\@IEEEgrabbedfirstoken=##1}% + \@IEEEgrabfirstoken#2\relax\relax\noindent + % \@IEEEgrabbedfirstoken has the first token, the rest are discarded + % if we need to put things into and hbox and go into math mode, do so now + \if@IEEEeqnarrayboxHBOXSW \leavevmode \hbox \bgroup $\fi% + % use the appropriate vbox type + \if\@IEEEgrabbedfirstoken t\relax\vtop\else\if\@IEEEgrabbedfirstoken c\relax% + \vcenter\else\vbox\fi\fi\bgroup% + \@IEEEeqnarrayISinnertrue% commands are now within the lines + \ifx#3\relax\halign\else\halign to #3\relax\fi% + \bgroup + % "exspand" the preamble + \span\the\@IEEEtrantmptoksA\cr} + +% carry strut status and enter the isolation/strut column, +% exit from math mode if needed, and exit +\def\end@IEEEeqnarraybox{\@IEEEeqnarrayglobalizestrutstatus% carry strut status +&% enter isolation/strut column +\@IEEEeqnarrayinsertstrut% do strut if needed +\@IEEEeqnarraymasterstrutrestore% restore the previous master strut values +% reset the strut system for next IEEEeqnarray +% (sets local strut values back to previous master strut values) +\@IEEEeqnarraystrutreset% +% ensure last line, exit from halign, close vbox +\crcr\egroup\egroup% +% exit from math mode and close hbox if needed +\if@IEEEeqnarrayboxHBOXSW $\egroup\fi} + + + +% IEEEeqnarraybox uses a modifed \\ instead of the plain \cr to +% end rows. This allows for things like \\[vskip amount] +% This "cr" macros are modified versions those for LaTeX2e's eqnarray +% For IEEEeqnarraybox, \\* is the same as \\ +% the {\ifnum0=`} braces must be kept away from the last column to avoid +% altering spacing of its math, so we use & to advance to the isolation/strut column +% carry strut status into isolation/strut column +\def\@IEEEeqnarrayboxcr{\@IEEEeqnarrayglobalizestrutstatus% carry strut status +&% enter isolation/strut column +\@IEEEeqnarrayinsertstrut% do strut if needed +% reset the strut system for next line or IEEEeqnarray +\@IEEEeqnarraystrutreset% +{\ifnum0=`}\fi% +\@ifstar{\@IEEEeqnarrayboxYCR}{\@IEEEeqnarrayboxYCR}} + +% test and setup the optional argument to \\[] +\def\@IEEEeqnarrayboxYCR{\@testopt\@IEEEeqnarrayboxXCR\z@skip} + +% IEEEeqnarraybox does not automatically increase line spacing by \jot +\def\@IEEEeqnarrayboxXCR[#1]{\ifnum0=`{\fi}% +\cr\noalign{\if@IEEEeqnarraystarform\else\vskip\jot\fi\vskip#1\relax}} + + + +% starts the halign preamble build +\def\@IEEEbuildpreamble{\@IEEEtrantmptoksA={}% clear token register +\let\@IEEEBPcurtype=u%current column type is not yet known +\let\@IEEEBPprevtype=s%the previous column type was the start +\let\@IEEEBPnexttype=u%next column type is not yet known +% ensure these are valid +\def\@IEEEBPcurglue={0pt plus 0pt minus 0pt}% +\def\@IEEEBPcurcolname{@IEEEdefault}% name of current column definition +% currently acquired numerically referenced glue +% use a name that is easier to remember +\let\@IEEEBPcurnum=\@IEEEtrantmpcountA% +\@IEEEBPcurnum=0% +% tracks number of columns in the preamble +\@IEEEeqnnumcols=0% +% record the default end glues +\edef\@IEEEBPstartglue{\@IEEEeqnarraycolSEPdefaultstart}% +\edef\@IEEEBPendglue{\@IEEEeqnarraycolSEPdefaultend}% +% now parse the user's column specifications +\@@IEEEbuildpreamble} + + +% parses and builds the halign preamble +\def\@@IEEEbuildpreamble#1#2{\let\@@nextIEEEbuildpreamble=\@@IEEEbuildpreamble% +% use only the very first token to check the end +% \noindent is used as a delimiter as \end can be present here +\def\@IEEEgrabfirstoken##1##2\noindent{\let\@IEEEgrabbedfirstoken=##1}% +\@IEEEgrabfirstoken#1\relax\relax\noindent +\ifx\@IEEEgrabbedfirstoken\end\let\@@nextIEEEbuildpreamble=\@@IEEEfinishpreamble\else% +% identify current and next token type +\@IEEEgetcoltype{#1}{\@IEEEBPcurtype}{1}% current, error on invalid +\@IEEEgetcoltype{#2}{\@IEEEBPnexttype}{0}% next, no error on invalid next +% if curtype is a glue, get the glue def +\if\@IEEEBPcurtype g\@IEEEgetcurglue{#1}{\@IEEEBPcurglue}\fi% +% if curtype is a column, get the column def and set the current column name +\if\@IEEEBPcurtype c\@IEEEgetcurcol{#1}\fi% +% if curtype is a numeral, acquire the user defined glue +\if\@IEEEBPcurtype n\@IEEEprocessNcol{#1}\fi% +% process the acquired glue +\if\@IEEEBPcurtype g\@IEEEprocessGcol\fi% +% process the acquired col +\if\@IEEEBPcurtype c\@IEEEprocessCcol\fi% +% ready prevtype for next col spec. +\let\@IEEEBPprevtype=\@IEEEBPcurtype% +% be sure and put back the future token(s) as a group +\fi\@@nextIEEEbuildpreamble{#2}} + + +% executed just after preamble build is completed +% warn about zero cols, and if prevtype type = u, put in end tabskip glue +\def\@@IEEEfinishpreamble#1{\ifnum\@IEEEeqnnumcols<1\relax +\@IEEEclspkgerror{No column specifiers declared for IEEEeqnarray}% +{At least one column type must be declared for each IEEEeqnarray.}% +\fi%num cols less than 1 +%if last type undefined, set default end tabskip glue +\if\@IEEEBPprevtype u\@IEEEappendtoksA{\tabskip=\@IEEEBPendglue}\fi} + + +% Identify and return the column specifier's type code +\def\@IEEEgetcoltype#1#2#3{% +% use only the very first token to determine the type +% \noindent is used as a delimiter as \end can be present here +\def\@IEEEgrabfirstoken##1##2\noindent{\let\@IEEEgrabbedfirstoken=##1}% +\@IEEEgrabfirstoken#1\relax\relax\noindent +% \@IEEEgrabfirstoken has the first token, the rest are discarded +% n = number +% g = glue (any other char in catagory 12) +% c = letter +% e = \end +% u = undefined +% third argument: 0 = no error message, 1 = error on invalid char +\let#2=u\relax% assume invalid until know otherwise +\ifx\@IEEEgrabbedfirstoken\end\let#2=e\else +\ifcat\@IEEEgrabbedfirstoken\relax\else% screen out control sequences +\if0\@IEEEgrabbedfirstoken\let#2=n\else +\if1\@IEEEgrabbedfirstoken\let#2=n\else +\if2\@IEEEgrabbedfirstoken\let#2=n\else +\if3\@IEEEgrabbedfirstoken\let#2=n\else +\if4\@IEEEgrabbedfirstoken\let#2=n\else +\if5\@IEEEgrabbedfirstoken\let#2=n\else +\if6\@IEEEgrabbedfirstoken\let#2=n\else +\if7\@IEEEgrabbedfirstoken\let#2=n\else +\if8\@IEEEgrabbedfirstoken\let#2=n\else +\if9\@IEEEgrabbedfirstoken\let#2=n\else +\ifcat,\@IEEEgrabbedfirstoken\let#2=g\relax +\else\ifcat a\@IEEEgrabbedfirstoken\let#2=c\relax\fi\fi\fi\fi\fi\fi\fi\fi\fi\fi\fi\fi\fi\fi +\if#2u\relax +\if0\noexpand#3\relax\else\@IEEEclspkgerror{Invalid character in column specifications}% +{Only letters, numerals and certain other symbols are allowed \MessageBreak +as IEEEeqnarray column specifiers.}\fi\fi} + + +% identify the current letter referenced column +% if invalid, use a default column +\def\@IEEEgetcurcol#1{\expandafter\ifx\csname @IEEEeqnarraycolDEF#1\endcsname\@IEEEeqnarraycolisdefined% +\def\@IEEEBPcurcolname{#1}\else% invalid column name +\@IEEEclspkgerror{Invalid column type "#1" in column specifications.\MessageBreak +Using a default centering column instead}% +{You must define IEEEeqnarray column types before use.}% +\def\@IEEEBPcurcolname{@IEEEdefault}\fi} + + +% identify and return the predefined (punctuation) glue value +\def\@IEEEgetcurglue#1#2{% +% ! = \! (neg small) -0.16667em (-3/18 em) +% , = \, (small) 0.16667em ( 3/18 em) +% : = \: (med) 0.22222em ( 4/18 em) +% ; = \; (large) 0.27778em ( 5/18 em) +% ' = \quad 1em +% " = \qquad 2em +% . = 0.5\arraycolsep +% / = \arraycolsep +% ? = 2\arraycolsep +% * = 1fil +% + = \@IEEEeqnarraycolSEPcenter +% - = \@IEEEeqnarraycolSEPzero +% Note that all em values are referenced to the math font (textfont2) fontdimen6 +% value for 1em. +% +% use only the very first token to determine the type +% this prevents errant tokens from getting in the main text +% \noindent is used as a delimiter here +\def\@IEEEgrabfirstoken##1##2\noindent{\let\@IEEEgrabbedfirstoken=##1}% +\@IEEEgrabfirstoken#1\relax\relax\noindent +% get the math font 1em value +% LaTeX2e's NFSS2 does not preload the fonts, but \IEEEeqnarray needs +% to gain access to the math (\textfont2) font's spacing parameters. +% So we create a bogus box here that uses the math font to ensure +% that \textfont2 is loaded and ready. If this is not done, +% the \textfont2 stuff here may not work. +% Thanks to Bernd Raichle for his 1997 post on this topic. +{\setbox0=\hbox{$\displaystyle\relax$}}% +% fontdimen6 has the width of 1em (a quad). +\@IEEEtrantmpdimenA=\fontdimen6\textfont2\relax% +% identify the glue value based on the first token +% we discard anything after the first +\if!\@IEEEgrabbedfirstoken\@IEEEtrantmpdimenA=-0.16667\@IEEEtrantmpdimenA\edef#2{\the\@IEEEtrantmpdimenA}\else +\if,\@IEEEgrabbedfirstoken\@IEEEtrantmpdimenA=0.16667\@IEEEtrantmpdimenA\edef#2{\the\@IEEEtrantmpdimenA}\else +\if:\@IEEEgrabbedfirstoken\@IEEEtrantmpdimenA=0.22222\@IEEEtrantmpdimenA\edef#2{\the\@IEEEtrantmpdimenA}\else +\if;\@IEEEgrabbedfirstoken\@IEEEtrantmpdimenA=0.27778\@IEEEtrantmpdimenA\edef#2{\the\@IEEEtrantmpdimenA}\else +\if'\@IEEEgrabbedfirstoken\@IEEEtrantmpdimenA=1\@IEEEtrantmpdimenA\edef#2{\the\@IEEEtrantmpdimenA}\else +\if"\@IEEEgrabbedfirstoken\@IEEEtrantmpdimenA=2\@IEEEtrantmpdimenA\edef#2{\the\@IEEEtrantmpdimenA}\else +\if.\@IEEEgrabbedfirstoken\@IEEEtrantmpdimenA=0.5\arraycolsep\edef#2{\the\@IEEEtrantmpdimenA}\else +\if/\@IEEEgrabbedfirstoken\edef#2{\the\arraycolsep}\else +\if?\@IEEEgrabbedfirstoken\@IEEEtrantmpdimenA=2\arraycolsep\edef#2{\the\@IEEEtrantmpdimenA}\else +\if *\@IEEEgrabbedfirstoken\edef#2{0pt plus 1fil minus 0pt}\else +\if+\@IEEEgrabbedfirstoken\edef#2{\@IEEEeqnarraycolSEPcenter}\else +\if-\@IEEEgrabbedfirstoken\edef#2{\@IEEEeqnarraycolSEPzero}\else +\edef#2{\@IEEEeqnarraycolSEPzero}% +\@IEEEclspkgerror{Invalid predefined inter-column glue type "#1" in\MessageBreak +column specifications. Using a default value of\MessageBreak +0pt instead}% +{Only !,:;'"./?*+ and - are valid predefined glue types in the\MessageBreak +IEEEeqnarray column specifications.}\fi\fi\fi\fi\fi\fi\fi\fi\fi\fi\fi\fi} + + + +% process a numerical digit from the column specification +% and look up the corresponding user defined glue value +% can transform current type from n to g or a as the user defined glue is acquired +\def\@IEEEprocessNcol#1{\if\@IEEEBPprevtype g% +\@IEEEclspkgerror{Back-to-back inter-column glue specifiers in column\MessageBreak +specifications. Ignoring consecutive glue specifiers\MessageBreak +after the first}% +{You cannot have two or more glue types next to each other\MessageBreak +in the IEEEeqnarray column specifications.}% +\let\@IEEEBPcurtype=a% abort this glue, future digits will be discarded +\@IEEEBPcurnum=0\relax% +\else% if we previously aborted a glue +\if\@IEEEBPprevtype a\@IEEEBPcurnum=0\let\@IEEEBPcurtype=a%maintain digit abortion +\else%acquire this number +% save the previous type before the numerical digits started +\if\@IEEEBPprevtype n\else\let\@IEEEBPprevsavedtype=\@IEEEBPprevtype\fi% +\multiply\@IEEEBPcurnum by 10\relax% +\advance\@IEEEBPcurnum by #1\relax% add in number, \relax is needed to stop TeX's number scan +\if\@IEEEBPnexttype n\else%close acquisition +\expandafter\ifx\csname @IEEEeqnarraycolSEPDEF\expandafter\romannumeral\number\@IEEEBPcurnum\endcsname\@IEEEeqnarraycolisdefined% +\edef\@IEEEBPcurglue{\csname @IEEEeqnarraycolSEP\expandafter\romannumeral\number\@IEEEBPcurnum\endcsname}% +\else%user glue not defined +\@IEEEclspkgerror{Invalid user defined inter-column glue type "\number\@IEEEBPcurnum" in\MessageBreak +column specifications. Using a default value of\MessageBreak +0pt instead}% +{You must define all IEEEeqnarray numerical inter-column glue types via\MessageBreak +\string\IEEEeqnarraydefcolsep \space before they are used in column specifications.}% +\edef\@IEEEBPcurglue{\@IEEEeqnarraycolSEPzero}% +\fi% glue defined or not +\let\@IEEEBPcurtype=g% change the type to reflect the acquired glue +\let\@IEEEBPprevtype=\@IEEEBPprevsavedtype% restore the prev type before this number glue +\@IEEEBPcurnum=0\relax%ready for next acquisition +\fi%close acquisition, get glue +\fi%discard or acquire number +\fi%prevtype glue or not +} + + +% process an acquired glue +% add any acquired column/glue pair to the preamble +\def\@IEEEprocessGcol{\if\@IEEEBPprevtype a\let\@IEEEBPcurtype=a%maintain previous glue abortions +\else +% if this is the start glue, save it, but do nothing else +% as this is not used in the preamble, but before +\if\@IEEEBPprevtype s\edef\@IEEEBPstartglue{\@IEEEBPcurglue}% +\else%not the start glue +\if\@IEEEBPprevtype g%ignore if back to back glues +\@IEEEclspkgerror{Back-to-back inter-column glue specifiers in column\MessageBreak +specifications. Ignoring consecutive glue specifiers\MessageBreak +after the first}% +{You cannot have two or more glue types next to each other\MessageBreak +in the IEEEeqnarray column specifications.}% +\let\@IEEEBPcurtype=a% abort this glue +\else% not a back to back glue +\if\@IEEEBPprevtype c\relax% if the previoustype was a col, add column/glue pair to preamble +\ifnum\@IEEEeqnnumcols>0\relax\@IEEEappendtoksA{&}\fi +\toks0={##}% +% make preamble advance col counter if this environment needs this +\if@advanceIEEEeqncolcnt\@IEEEappendtoksA{\global\advance\@IEEEeqncolcnt by 1\relax}\fi +% insert the column defintion into the preamble, being careful not to expand +% the column definition +\@IEEEappendtoksA{\tabskip=\@IEEEBPcurglue}% +\@IEEEappendNOEXPANDtoksA{\begingroup\csname @IEEEeqnarraycolPRE}% +\@IEEEappendtoksA{\@IEEEBPcurcolname}% +\@IEEEappendNOEXPANDtoksA{\endcsname}% +\@IEEEappendtoksA{\the\toks0}% +\@IEEEappendNOEXPANDtoksA{\relax\relax\relax\relax\relax% +\relax\relax\relax\relax\relax\csname @IEEEeqnarraycolPOST}% +\@IEEEappendtoksA{\@IEEEBPcurcolname}% +\@IEEEappendNOEXPANDtoksA{\endcsname\relax\relax\relax\relax\relax% +\relax\relax\relax\relax\relax\endgroup}% +\advance\@IEEEeqnnumcols by 1\relax%one more column in the preamble +\else% error: non-start glue with no pending column +\@IEEEclspkgerror{Inter-column glue specifier without a prior column\MessageBreak +type in the column specifications. Ignoring this glue\MessageBreak +specifier}% +{Except for the first and last positions, glue can be placed only\MessageBreak +between column types.}% +\let\@IEEEBPcurtype=a% abort this glue +\fi% previous was a column +\fi% back-to-back glues +\fi% is start column glue +\fi% prev type not a +} + + +% process an acquired letter referenced column and, if necessary, add it to the preamble +\def\@IEEEprocessCcol{\if\@IEEEBPnexttype g\else +\if\@IEEEBPnexttype n\else +% we have a column followed by something other than a glue (or numeral glue) +% so we must add this column to the preamble now +\ifnum\@IEEEeqnnumcols>0\relax\@IEEEappendtoksA{&}\fi%col separator for those after the first +\if\@IEEEBPnexttype e\@IEEEappendtoksA{\tabskip=\@IEEEBPendglue\relax}\else%put in end glue +\@IEEEappendtoksA{\tabskip=\@IEEEeqnarraycolSEPdefaultmid\relax}\fi% or default mid glue +\toks0={##}% +% make preamble advance col counter if this environment needs this +\if@advanceIEEEeqncolcnt\@IEEEappendtoksA{\global\advance\@IEEEeqncolcnt by 1\relax}\fi +% insert the column definition into the preamble, being careful not to expand +% the column definition +\@IEEEappendNOEXPANDtoksA{\begingroup\csname @IEEEeqnarraycolPRE}% +\@IEEEappendtoksA{\@IEEEBPcurcolname}% +\@IEEEappendNOEXPANDtoksA{\endcsname}% +\@IEEEappendtoksA{\the\toks0}% +\@IEEEappendNOEXPANDtoksA{\relax\relax\relax\relax\relax% +\relax\relax\relax\relax\relax\csname @IEEEeqnarraycolPOST}% +\@IEEEappendtoksA{\@IEEEBPcurcolname}% +\@IEEEappendNOEXPANDtoksA{\endcsname\relax\relax\relax\relax\relax% +\relax\relax\relax\relax\relax\endgroup}% +\advance\@IEEEeqnnumcols by 1\relax%one more column in the preamble +\fi%next type not numeral +\fi%next type not glue +} + + +%% +%% END OF IEEEeqnarry DEFINITIONS +%% + + + + +% set up the running headings, this complex because of all the different +% modes IEEEtran supports +\if@twoside + \ifCLASSOPTIONtechnote + \def\ps@headings{% + \def\@oddhead{\hbox{}\scriptsize\leftmark \hfil \thepage} + \def\@evenhead{\scriptsize\thepage \hfil \leftmark\hbox{}} + \ifCLASSOPTIONdraftcls + \ifCLASSOPTIONdraftclsnofoot + \def\@oddfoot{}\def\@evenfoot{}% + \else + \def\@oddfoot{\scriptsize\@date\hfil DRAFT} + \def\@evenfoot{\scriptsize DRAFT\hfil\@date} + \fi + \else + \def\@oddfoot{}\def\@evenfoot{} + \fi} + \else % not a technote + \def\ps@headings{% + \ifCLASSOPTIONconference + \def\@oddhead{} + \def\@evenhead{} + \else + \def\@oddhead{\hbox{}\scriptsize\rightmark \hfil \thepage} + \def\@evenhead{\scriptsize\thepage \hfil \leftmark\hbox{}} + \fi + \ifCLASSOPTIONdraftcls + \def\@oddhead{\hbox{}\scriptsize\rightmark \hfil \thepage} + \def\@evenhead{\scriptsize\thepage \hfil \leftmark\hbox{}} + \ifCLASSOPTIONdraftclsnofoot + \def\@oddfoot{}\def\@evenfoot{}% + \else + \def\@oddfoot{\scriptsize\@date\hfil DRAFT} + \def\@evenfoot{\scriptsize DRAFT\hfil\@date} + \fi + \else + \def\@oddfoot{}\def\@evenfoot{}% + \fi} + \fi +\else % single side +\def\ps@headings{% + \ifCLASSOPTIONconference + \def\@oddhead{} + \def\@evenhead{} + \else + \def\@oddhead{\hbox{}\scriptsize\leftmark \hfil \thepage} + \def\@evenhead{} + \fi + \ifCLASSOPTIONdraftcls + \def\@oddhead{\hbox{}\scriptsize\leftmark \hfil \thepage} + \def\@evenhead{} + \ifCLASSOPTIONdraftclsnofoot + \def\@oddfoot{} + \else + \def\@oddfoot{\scriptsize \@date \hfil DRAFT} + \fi + \else + \def\@oddfoot{} + \fi + \def\@evenfoot{}} +\fi + + +% title page style +\def\ps@IEEEtitlepagestyle{\def\@oddfoot{}\def\@evenfoot{}% +\ifCLASSOPTIONconference + \def\@oddhead{}% + \def\@evenhead{}% +\else + \def\@oddhead{\hbox{}\scriptsize\leftmark \hfil \thepage}% + \def\@evenhead{\scriptsize\thepage \hfil \leftmark\hbox{}}% +\fi +\ifCLASSOPTIONdraftcls + \def\@oddhead{\hbox{}\scriptsize\leftmark \hfil \thepage}% + \def\@evenhead{\scriptsize\thepage \hfil \leftmark\hbox{}}% + \ifCLASSOPTIONdraftclsnofoot\else + \def\@oddfoot{\scriptsize \@date\hfil DRAFT}% + \def\@evenfoot{\scriptsize DRAFT\hfil \@date}% + \fi +\else + % all non-draft mode footers + \if@IEEEusingpubid + % for title pages that are using a pubid + % do not repeat pubid if using peer review option + \ifCLASSOPTIONpeerreview + \else + \footskip 0pt% + \ifCLASSOPTIONcompsoc + \def\@oddfoot{\hss\normalfont\scriptsize\raisebox{-1.5\@IEEEnormalsizeunitybaselineskip}[0ex][0ex]{\@IEEEpubid}\hss}% + \def\@evenfoot{\hss\normalfont\scriptsize\raisebox{-1.5\@IEEEnormalsizeunitybaselineskip}[0ex][0ex]{\@IEEEpubid}\hss}% + \else + \def\@oddfoot{\hss\normalfont\footnotesize\raisebox{1.5ex}[1.5ex]{\@IEEEpubid}\hss}% + \def\@evenfoot{\hss\normalfont\footnotesize\raisebox{1.5ex}[1.5ex]{\@IEEEpubid}\hss}% + \fi + \fi + \fi +\fi} + + +% peer review cover page style +\def\ps@IEEEpeerreviewcoverpagestyle{% +\def\@oddhead{}\def\@evenhead{}% +\def\@oddfoot{}\def\@evenfoot{}% +\ifCLASSOPTIONdraftcls + \ifCLASSOPTIONdraftclsnofoot\else + \def\@oddfoot{\scriptsize \@date\hfil DRAFT}% + \def\@evenfoot{\scriptsize DRAFT\hfil \@date}% + \fi +\else + % non-draft mode footers + \if@IEEEusingpubid + \footskip 0pt% + \ifCLASSOPTIONcompsoc + \def\@oddfoot{\hss\normalfont\scriptsize\raisebox{-1.5\@IEEEnormalsizeunitybaselineskip}[0ex][0ex]{\@IEEEpubid}\hss}% + \def\@evenfoot{\hss\normalfont\scriptsize\raisebox{-1.5\@IEEEnormalsizeunitybaselineskip}[0ex][0ex]{\@IEEEpubid}\hss}% + \else + \def\@oddfoot{\hss\normalfont\footnotesize\raisebox{1.5ex}[1.5ex]{\@IEEEpubid}\hss}% + \def\@evenfoot{\hss\normalfont\footnotesize\raisebox{1.5ex}[1.5ex]{\@IEEEpubid}\hss}% + \fi + \fi +\fi} + + +% start with empty headings +\def\rightmark{}\def\leftmark{} + + +%% Defines the command for putting the header. \footernote{TEXT} is the same +%% as \markboth{TEXT}{TEXT}. +%% Note that all the text is forced into uppercase, if you have some text +%% that needs to be in lower case, for instance et. al., then either manually +%% set \leftmark and \rightmark or use \MakeLowercase{et. al.} within the +%% arguments to \markboth. +\def\markboth#1#2{\def\leftmark{\@IEEEcompsoconly{\sffamily}\MakeUppercase{#1}}% +\def\rightmark{\@IEEEcompsoconly{\sffamily}\MakeUppercase{#2}}} +\def\footernote#1{\markboth{#1}{#1}} + +\def\today{\ifcase\month\or + January\or February\or March\or April\or May\or June\or + July\or August\or September\or October\or November\or December\fi + \space\number\day, \number\year} + + + + +%% CITATION AND BIBLIOGRAPHY COMMANDS +%% +%% V1.6 no longer supports the older, nonstandard \shortcite and \citename setup stuff +% +% +% Modify Latex2e \@citex to separate citations with "], [" +\def\@citex[#1]#2{% + \let\@citea\@empty + \@cite{\@for\@citeb:=#2\do + {\@citea\def\@citea{], [}% + \edef\@citeb{\expandafter\@firstofone\@citeb\@empty}% + \if@filesw\immediate\write\@auxout{\string\citation{\@citeb}}\fi + \@ifundefined{b@\@citeb}{\mbox{\reset@font\bfseries ?}% + \G@refundefinedtrue + \@latex@warning + {Citation `\@citeb' on page \thepage \space undefined}}% + {\hbox{\csname b@\@citeb\endcsname}}}}{#1}} + +% V1.6 we create hooks for the optional use of Donald Arseneau's +% cite.sty package. cite.sty is "smart" and will notice that the +% following format controls are already defined and will not +% redefine them. The result will be the proper sorting of the +% citation numbers and auto detection of 3 or more entry "ranges" - +% all in IEEE style: [1], [2], [5]--[7], [12] +% This also allows for an optional note, i.e., \cite[mynote]{..}. +% If the \cite with note has more than one reference, the note will +% be applied to the last of the listed references. It is generally +% desired that if a note is given, only one reference is listed in +% that \cite. +% Thanks to Mr. Arseneau for providing the required format arguments +% to produce the IEEE style. +\def\citepunct{], [} +\def\citedash{]--[} + +% V1.7 default to using same font for urls made by url.sty +\AtBeginDocument{\csname url@samestyle\endcsname} + +% V1.6 class files should always provide these +\def\newblock{\hskip .11em\@plus.33em\@minus.07em} +\let\@openbib@code\@empty + + +% Provide support for the control entries of IEEEtran.bst V1.00 and later. +% V1.7 optional argument allows for a different aux file to be specified in +% order to handle multiple bibliographies. For example, with multibib.sty: +% \newcites{sec}{Secondary Literature} +% \bstctlcite[@auxoutsec]{BSTcontrolhak} +\def\bstctlcite{\@ifnextchar[{\@bstctlcite}{\@bstctlcite[@auxout]}} +\def\@bstctlcite[#1]#2{\@bsphack + \@for\@citeb:=#2\do{% + \edef\@citeb{\expandafter\@firstofone\@citeb}% + \if@filesw\immediate\write\csname #1\endcsname{\string\citation{\@citeb}}\fi}% + \@esphack} + +% V1.6 provide a way for a user to execute a command just before +% a given reference number - used to insert a \newpage to balance +% the columns on the last page +\edef\@IEEEtriggerrefnum{0} % the default of zero means that + % the command is not executed +\def\@IEEEtriggercmd{\newpage} + +% allow the user to alter the triggered command +\long\def\IEEEtriggercmd#1{\long\def\@IEEEtriggercmd{#1}} + +% allow user a way to specify the reference number just before the +% command is executed +\def\IEEEtriggeratref#1{\@IEEEtrantmpcountA=#1% +\edef\@IEEEtriggerrefnum{\the\@IEEEtrantmpcountA}}% + +% trigger command at the given reference +\def\@IEEEbibitemprefix{\@IEEEtrantmpcountA=\@IEEEtriggerrefnum\relax% +\advance\@IEEEtrantmpcountA by -1\relax% +\ifnum\c@enumiv=\@IEEEtrantmpcountA\relax\@IEEEtriggercmd\relax\fi} + + +\def\@biblabel#1{[#1]} + +% compsoc journals left align the reference numbers +\@IEEEcompsocnotconfonly{\def\@biblabel#1{[#1]\hfill}} + +% controls bib item spacing +\def\IEEEbibitemsep{2.5pt plus .5pt} + +\@IEEEcompsocconfonly{\def\IEEEbibitemsep{1\baselineskip plus 0.25\baselineskip minus 0.25\baselineskip}} + + +\def\thebibliography#1{\section*{\refname}% + \addcontentsline{toc}{section}{\refname}% + % V1.6 add some rubber space here and provide a command trigger + \footnotesize\@IEEEcompsocconfonly{\small}\vskip 0.3\baselineskip plus 0.1\baselineskip minus 0.1\baselineskip% + \list{\@biblabel{\@arabic\c@enumiv}}% + {\settowidth\labelwidth{\@biblabel{#1}}% + \leftmargin\labelwidth + \labelsep 1em + \advance\leftmargin\labelsep\relax + \itemsep \IEEEbibitemsep\relax + \usecounter{enumiv}% + \let\p@enumiv\@empty + \renewcommand\theenumiv{\@arabic\c@enumiv}}% + \let\@IEEElatexbibitem\bibitem% + \def\bibitem{\@IEEEbibitemprefix\@IEEElatexbibitem}% +\def\newblock{\hskip .11em plus .33em minus .07em}% +% originally: +% \sloppy\clubpenalty4000\widowpenalty4000% +% by adding the \interlinepenalty here, we make it more +% difficult, but not impossible, for LaTeX to break within a reference. +% IEEE almost never breaks a reference (but they do it more often with +% technotes). You may get an underfull vbox warning around the bibliography, +% but the final result will be much more like what IEEE will publish. +% MDS 11/2000 +\ifCLASSOPTIONtechnote\sloppy\clubpenalty4000\widowpenalty4000\interlinepenalty100% +\else\sloppy\clubpenalty4000\widowpenalty4000\interlinepenalty500\fi% + \sfcode`\.=1000\relax} +\let\endthebibliography=\endlist + + + + +% TITLE PAGE COMMANDS +% +% +% \IEEEmembership is used to produce the sublargesize italic font used to indicate author +% IEEE membership. compsoc uses a large size sans slant font +\def\IEEEmembership#1{{\@IEEEnotcompsoconly{\sublargesize}\normalfont\@IEEEcompsoconly{\sffamily}\textit{#1}}} + + +% \IEEEauthorrefmark{} produces a footnote type symbol to indicate author affiliation. +% When given an argument of 1 to 9, \IEEEauthorrefmark{} follows the standard LaTeX footnote +% symbol sequence convention. However, for arguments 10 and above, \IEEEauthorrefmark{} +% reverts to using lower case roman numerals, so it cannot overflow. Do note that you +% cannot use \footnotemark[] in place of \IEEEauthorrefmark{} within \author as the footnote +% symbols will have been turned off to prevent \thanks from creating footnote marks. +% \IEEEauthorrefmark{} produces a symbol that appears to LaTeX as having zero vertical +% height - this allows for a more compact line packing, but the user must ensure that +% the interline spacing is large enough to prevent \IEEEauthorrefmark{} from colliding +% with the text above. +% V1.7 make this a robust command +\DeclareRobustCommand*{\IEEEauthorrefmark}[1]{\raisebox{0pt}[0pt][0pt]{\textsuperscript{\footnotesize\ensuremath{\ifcase#1\or *\or \dagger\or \ddagger\or% + \mathsection\or \mathparagraph\or \|\or **\or \dagger\dagger% + \or \ddagger\ddagger \else\textsuperscript{\expandafter\romannumeral#1}\fi}}}} + + +% FONT CONTROLS AND SPACINGS FOR CONFERENCE MODE AUTHOR NAME AND AFFILIATION BLOCKS +% +% The default font styles for the author name and affiliation blocks (confmode) +\def\@IEEEauthorblockNstyle{\normalfont\@IEEEcompsocnotconfonly{\sffamily}\sublargesize\@IEEEcompsocconfonly{\large}} +\def\@IEEEauthorblockAstyle{\normalfont\@IEEEcompsocnotconfonly{\sffamily}\@IEEEcompsocconfonly{\itshape}\normalsize\@IEEEcompsocconfonly{\large}} +% The default if the user does not use an author block +\def\@IEEEauthordefaulttextstyle{\normalfont\@IEEEcompsocnotconfonly{\sffamily}\sublargesize} + +% spacing from title (or special paper notice) to author name blocks (confmode) +% can be negative +\def\@IEEEauthorblockconfadjspace{-0.25em} +% compsoc conferences need more space here +\@IEEEcompsocconfonly{\def\@IEEEauthorblockconfadjspace{0.75\@IEEEnormalsizeunitybaselineskip}} +\ifCLASSOPTIONconference\def\@IEEEauthorblockconfadjspace{20pt}\fi + +% spacing between name and affiliation blocks (confmode) +% This can be negative. +% IEEE doesn't want any added spacing here, but I will leave these +% controls in place in case they ever change their mind. +% Personally, I like 0.75ex. +%\def\@IEEEauthorblockNtopspace{0.75ex} +%\def\@IEEEauthorblockAtopspace{0.75ex} +\def\@IEEEauthorblockNtopspace{0.0ex} +\def\@IEEEauthorblockAtopspace{0.0ex} +% baseline spacing within name and affiliation blocks (confmode) +% must be positive, spacings below certain values will make +% the position of line of text sensitive to the contents of the +% line above it i.e., whether or not the prior line has descenders, +% subscripts, etc. For this reason it is a good idea to keep +% these above 2.6ex +\def\@IEEEauthorblockNinterlinespace{2.6ex} +\def\@IEEEauthorblockAinterlinespace{2.75ex} + +% This tracks the required strut size. +% See the \@IEEEauthorhalign command for the actual default value used. +\def\@IEEEauthorblockXinterlinespace{2.7ex} + +% variables to retain font size and style across groups +% values given here have no effect as they will be overwritten later +\gdef\@IEEESAVESTATEfontsize{10} +\gdef\@IEEESAVESTATEfontbaselineskip{12} +\gdef\@IEEESAVESTATEfontencoding{OT1} +\gdef\@IEEESAVESTATEfontfamily{ptm} +\gdef\@IEEESAVESTATEfontseries{m} +\gdef\@IEEESAVESTATEfontshape{n} + +% saves the current font attributes +\def\@IEEEcurfontSAVE{\global\let\@IEEESAVESTATEfontsize\f@size% +\global\let\@IEEESAVESTATEfontbaselineskip\f@baselineskip% +\global\let\@IEEESAVESTATEfontencoding\f@encoding% +\global\let\@IEEESAVESTATEfontfamily\f@family% +\global\let\@IEEESAVESTATEfontseries\f@series% +\global\let\@IEEESAVESTATEfontshape\f@shape} + +% restores the saved font attributes +\def\@IEEEcurfontRESTORE{\fontsize{\@IEEESAVESTATEfontsize}{\@IEEESAVESTATEfontbaselineskip}% +\fontencoding{\@IEEESAVESTATEfontencoding}% +\fontfamily{\@IEEESAVESTATEfontfamily}% +\fontseries{\@IEEESAVESTATEfontseries}% +\fontshape{\@IEEESAVESTATEfontshape}% +\selectfont} + + +% variable to indicate if the current block is the first block in the column +\newif\if@IEEEprevauthorblockincol \@IEEEprevauthorblockincolfalse + + +% the command places a strut with height and depth = \@IEEEauthorblockXinterlinespace +% we use this technique to have complete manual control over the spacing of the lines +% within the halign environment. +% We set the below baseline portion at 30%, the above +% baseline portion at 70% of the total length. +% Responds to changes in the document's \baselinestretch +\def\@IEEEauthorstrutrule{\@IEEEtrantmpdimenA\@IEEEauthorblockXinterlinespace% +\@IEEEtrantmpdimenA=\baselinestretch\@IEEEtrantmpdimenA% +\rule[-0.3\@IEEEtrantmpdimenA]{0pt}{\@IEEEtrantmpdimenA}} + + +% blocks to hold the authors' names and affilations. +% Makes formatting easy for conferences +% +% use real definitions in conference mode +% name block +\def\IEEEauthorblockN#1{\relax\@IEEEauthorblockNstyle% set the default text style +\gdef\@IEEEauthorblockXinterlinespace{0pt}% disable strut for spacer row +% the \expandafter hides the \cr in conditional tex, see the array.sty docs +% for details, probably not needed here as the \cr is in a macro +% do a spacer row if needed +\if@IEEEprevauthorblockincol\expandafter\@IEEEauthorblockNtopspaceline\fi +\global\@IEEEprevauthorblockincoltrue% we now have a block in this column +%restore the correct strut value +\gdef\@IEEEauthorblockXinterlinespace{\@IEEEauthorblockNinterlinespace}% +% input the author names +#1% +% end the row if the user did not already +\crcr} +% spacer row for names +\def\@IEEEauthorblockNtopspaceline{\cr\noalign{\vskip\@IEEEauthorblockNtopspace}} +% +% affiliation block +\def\IEEEauthorblockA#1{\relax\@IEEEauthorblockAstyle% set the default text style +\gdef\@IEEEauthorblockXinterlinespace{0pt}%disable strut for spacer row +% the \expandafter hides the \cr in conditional tex, see the array.sty docs +% for details, probably not needed here as the \cr is in a macro +% do a spacer row if needed +\if@IEEEprevauthorblockincol\expandafter\@IEEEauthorblockAtopspaceline\fi +\global\@IEEEprevauthorblockincoltrue% we now have a block in this column +%restore the correct strut value +\gdef\@IEEEauthorblockXinterlinespace{\@IEEEauthorblockAinterlinespace}% +% input the author affiliations +#1% +% end the row if the user did not already +\crcr} +% spacer row for affiliations +\def\@IEEEauthorblockAtopspaceline{\cr\noalign{\vskip\@IEEEauthorblockAtopspace}} + + +% allow papers to compile even if author blocks are used in modes other +% than conference or peerreviewca. For such cases, we provide dummy blocks. +\ifCLASSOPTIONconference +\else + \ifCLASSOPTIONpeerreviewca\else + % not conference or peerreviewca mode + \def\IEEEauthorblockN#1{#1}% + \def\IEEEauthorblockA#1{#1}% + \fi +\fi + + + +% we provide our own halign so as not to have to depend on tabular +\def\@IEEEauthorhalign{\@IEEEauthordefaulttextstyle% default text style + \lineskip=0pt\relax% disable line spacing + \lineskiplimit=0pt\relax% + \baselineskip=0pt\relax% + \@IEEEcurfontSAVE% save the current font + \mathsurround\z@\relax% no extra spacing around math + \let\\\@IEEEauthorhaligncr% replace newline with halign friendly one + \tabskip=0pt\relax% no column spacing + \everycr{}% ensure no problems here + \@IEEEprevauthorblockincolfalse% no author blocks yet + \def\@IEEEauthorblockXinterlinespace{2.7ex}% default interline space + \vtop\bgroup%vtop box + \halign\bgroup&\relax\hfil\@IEEEcurfontRESTORE\relax ##\relax + \hfil\@IEEEcurfontSAVE\@IEEEauthorstrutrule\cr} + +% ensure last line, exit from halign, close vbox +\def\end@IEEEauthorhalign{\crcr\egroup\egroup} + +% handle bogus star form +\def\@IEEEauthorhaligncr{{\ifnum0=`}\fi\@ifstar{\@@IEEEauthorhaligncr}{\@@IEEEauthorhaligncr}} + +% test and setup the optional argument to \\[] +\def\@@IEEEauthorhaligncr{\@testopt\@@@IEEEauthorhaligncr\z@skip} + +% end the line and do the optional spacer +\def\@@@IEEEauthorhaligncr[#1]{\ifnum0=`{\fi}\cr\noalign{\vskip#1\relax}} + + + +% flag to prevent multiple \and warning messages +\newif\if@IEEEWARNand +\@IEEEWARNandtrue + +% if in conference or peerreviewca modes, we support the use of \and as \author is a +% tabular environment, otherwise we warn the user that \and is invalid +% outside of conference or peerreviewca modes. +\def\and{\relax} % provide a bogus \and that we will then override + +\renewcommand{\and}[1][\relax]{\if@IEEEWARNand\typeout{** WARNING: \noexpand\and is valid only + when in conference or peerreviewca}\typeout{modes (line \the\inputlineno).}\fi\global\@IEEEWARNandfalse} + +\ifCLASSOPTIONconference% +\renewcommand{\and}[1][\hfill]{\end{@IEEEauthorhalign}#1\begin{@IEEEauthorhalign}}% +\fi +\ifCLASSOPTIONpeerreviewca +\renewcommand{\and}[1][\hfill]{\end{@IEEEauthorhalign}#1\begin{@IEEEauthorhalign}}% +\fi + + +% page clearing command +% based on LaTeX2e's \cleardoublepage, but allows different page styles +% for the inserted blank pages +\def\@IEEEcleardoublepage#1{\clearpage\if@twoside\ifodd\c@page\else +\hbox{}\thispagestyle{#1}\newpage\if@twocolumn\hbox{}\thispagestyle{#1}\newpage\fi\fi\fi} + + +% user command to invoke the title page +\def\maketitle{\par% + \begingroup% + \normalfont% + \def\thefootnote{}% the \thanks{} mark type is empty + \def\footnotemark{}% and kill space from \thanks within author + \let\@makefnmark\relax% V1.7, must *really* kill footnotemark to remove all \textsuperscript spacing as well. + \footnotesize% equal spacing between thanks lines + \footnotesep 0.7\baselineskip%see global setting of \footnotesep for more info + % V1.7 disable \thanks note indention for compsoc + \@IEEEcompsoconly{\long\def\@makefntext##1{\parindent 1em\noindent\hbox{\@makefnmark}##1}}% + \normalsize% + \ifCLASSOPTIONpeerreview + \newpage\global\@topnum\z@ \@maketitle\@IEEEstatictitlevskip\@IEEEaftertitletext% + \thispagestyle{IEEEpeerreviewcoverpagestyle}\@thanks% + \else + \if@twocolumn% + \ifCLASSOPTIONtechnote% + \newpage\global\@topnum\z@ \@maketitle\@IEEEstatictitlevskip\@IEEEaftertitletext% + \else + \twocolumn[\@maketitle\@IEEEstatictitlevskip\@IEEEaftertitletext]% + \fi + \else + \newpage\global\@topnum\z@ \@maketitle\@IEEEstatictitlevskip\@IEEEaftertitletext% + \fi + \thispagestyle{IEEEtitlepagestyle}\@thanks% + \fi + % pullup page for pubid if used. + \if@IEEEusingpubid + \enlargethispage{-\@IEEEpubidpullup}% + \fi + \endgroup + \setcounter{footnote}{0}\let\maketitle\relax\let\@maketitle\relax + \gdef\@thanks{}% + % v1.6b do not clear these as we will need the title again for peer review papers + % \gdef\@author{}\gdef\@title{}% + \let\thanks\relax} + + + +% V1.7 parbox to format \@IEEEcompsoctitleabstractindextext +\long\def\@IEEEcompsoctitleabstractindextextbox#1{\parbox{0.915\textwidth}{#1}} + +% formats the Title, authors names, affiliations and special paper notice +% THIS IS A CONTROLLED SPACING COMMAND! Do not allow blank lines or unintentional +% spaces to enter the definition - use % at the end of each line +\def\@maketitle{\newpage +\begingroup\centering +\ifCLASSOPTIONtechnote% technotes + {\bfseries\large\@IEEEcompsoconly{\sffamily}\@title\par}\vskip 1.3em{\lineskip .5em\@IEEEcompsoconly{\sffamily}\@author + \@IEEEspecialpapernotice\par{\@IEEEcompsoconly{\vskip 1.5em\relax + \@IEEEcompsoctitleabstractindextextbox{\@IEEEcompsoctitleabstractindextext}\par + \hfill\@IEEEcompsocdiamondline\hfill\hbox{}\par}}}\relax +\else% not a technote + \vskip0.2em{\Huge\@IEEEcompsoconly{\sffamily}\@IEEEcompsocconfonly{\normalfont\normalsize\vskip 2\@IEEEnormalsizeunitybaselineskip + \bfseries\Large}\@title\par}\vskip1.0em\par% + % V1.6 handle \author differently if in conference mode + \ifCLASSOPTIONconference% + {\@IEEEspecialpapernotice\mbox{}\vskip\@IEEEauthorblockconfadjspace% + \mbox{}\hfill\begin{@IEEEauthorhalign}\@author\end{@IEEEauthorhalign}\hfill\mbox{}\par}\relax + \else% peerreviewca, peerreview or journal + \ifCLASSOPTIONpeerreviewca + % peerreviewca handles author names just like conference mode + {\@IEEEcompsoconly{\sffamily}\@IEEEspecialpapernotice\mbox{}\vskip\@IEEEauthorblockconfadjspace% + \mbox{}\hfill\begin{@IEEEauthorhalign}\@author\end{@IEEEauthorhalign}\hfill\mbox{}\par + {\@IEEEcompsoconly{\vskip 1.5em\relax + \@IEEEcompsoctitleabstractindextextbox{\@IEEEcompsoctitleabstractindextext}\par\hfill + \@IEEEcompsocdiamondline\hfill\hbox{}\par}}}\relax + \else% journal or peerreview + {\lineskip.5em\@IEEEcompsoconly{\sffamily}\sublargesize\@author\@IEEEspecialpapernotice\par + {\@IEEEcompsoconly{\vskip 1.5em\relax + \@IEEEcompsoctitleabstractindextextbox{\@IEEEcompsoctitleabstractindextext}\par\hfill + \@IEEEcompsocdiamondline\hfill\hbox{}\par}}}\relax + \fi + \fi +\fi\par\endgroup} + + + +% V1.7 Computer Society "diamond line" which follows index terms for nonconference papers +\def\@IEEEcompsocdiamondline{\vrule depth 0pt height 0.5pt width 4cm\hspace{7.5pt}% +\raisebox{-3.5pt}{\fontfamily{pzd}\fontencoding{U}\fontseries{m}\fontshape{n}\fontsize{11}{12}\selectfont\char70}% +\hspace{7.5pt}\vrule depth 0pt height 0.5pt width 4cm\relax} + +% V1.7 standard LateX2e \thanks, but with \itshape under compsoc. Also make it a \long\def +% We also need to trigger the one-shot footnote rule +\def\@IEEEtriggeroneshotfootnoterule{\global\@IEEEenableoneshotfootnoteruletrue} + + +\long\def\thanks#1{\footnotemark + \protected@xdef\@thanks{\@thanks + \protect\footnotetext[\the\c@footnote]{\@IEEEcompsoconly{\itshape + \protect\@IEEEtriggeroneshotfootnoterule\relax}\ignorespaces#1}}} +\let\@thanks\@empty + +% V1.7 allow \author to contain \par's. This is needed to allow \thanks to contain \par. +\long\def\author#1{\gdef\@author{#1}} + + +% in addition to setting up IEEEitemize, we need to remove a baselineskip space above and +% below it because \list's \pars introduce blank lines because of the footnote struts. +\def\@IEEEsetupcompsocitemizelist{\def\labelitemi{$\bullet$}% +\setlength{\IEEElabelindent}{0pt}\setlength{\parskip}{0pt}% +\setlength{\partopsep}{0pt}\setlength{\topsep}{0.5\baselineskip}\vspace{-1\baselineskip}\relax} + + +% flag for fake non-compsoc \IEEEcompsocthanksitem - prevents line break on very first item +\newif\if@IEEEbreakcompsocthanksitem \@IEEEbreakcompsocthanksitemfalse + +\ifCLASSOPTIONcompsoc +% V1.7 compsoc bullet item \thanks +% also, we need to redefine this to destroy the argument in \@IEEEdynamictitlevspace +\long\def\IEEEcompsocitemizethanks#1{\relax\@IEEEbreakcompsocthanksitemfalse\footnotemark + \protected@xdef\@thanks{\@thanks + \protect\footnotetext[\the\c@footnote]{\itshape\protect\@IEEEtriggeroneshotfootnoterule + {\let\IEEEiedlistdecl\relax\protect\begin{IEEEitemize}[\protect\@IEEEsetupcompsocitemizelist]\ignorespaces#1\relax + \protect\end{IEEEitemize}}\protect\vspace{-1\baselineskip}}}} +\DeclareRobustCommand*{\IEEEcompsocthanksitem}{\item} +\else +% non-compsoc, allow for dual compilation via rerouting to normal \thanks +\long\def\IEEEcompsocitemizethanks#1{\thanks{#1}} +% redirect to "pseudo-par" \hfil\break\indent after swallowing [] from \IEEEcompsocthanksitem[] +\DeclareRobustCommand{\IEEEcompsocthanksitem}{\@ifnextchar [{\@IEEEthanksswallowoptionalarg}% +{\@IEEEthanksswallowoptionalarg[\relax]}} +% be sure and break only after first item, be sure and ignore spaces after optional argument +\def\@IEEEthanksswallowoptionalarg[#1]{\relax\if@IEEEbreakcompsocthanksitem\hfil\break +\indent\fi\@IEEEbreakcompsocthanksitemtrue\ignorespaces} +\fi + + +% V1.6b define the \IEEEpeerreviewmaketitle as needed +\ifCLASSOPTIONpeerreview +\def\IEEEpeerreviewmaketitle{\@IEEEcleardoublepage{empty}% +\ifCLASSOPTIONtwocolumn +\twocolumn[\@IEEEpeerreviewmaketitle\@IEEEdynamictitlevspace] +\else +\newpage\@IEEEpeerreviewmaketitle\@IEEEstatictitlevskip +\fi +\thispagestyle{IEEEtitlepagestyle}} +\else +% \IEEEpeerreviewmaketitle does nothing if peer review option has not been selected +\def\IEEEpeerreviewmaketitle{\relax} +\fi + +% peerreview formats the repeated title like the title in journal papers. +\def\@IEEEpeerreviewmaketitle{\begin{center}\@IEEEcompsoconly{\sffamily}% +\normalfont\normalsize\vskip0.2em{\Huge\@title\par}\vskip1.0em\par +\end{center}} + + + +% V1.6 +% this is a static rubber spacer between the title/authors and the main text +% used for single column text, or when the title appears in the first column +% of two column text (technotes). +\def\@IEEEstatictitlevskip{{\normalfont\normalsize +% adjust spacing to next text +% v1.6b handle peer review papers +\ifCLASSOPTIONpeerreview +% for peer review papers, the same value is used for both title pages +% regardless of the other paper modes + \vskip 1\baselineskip plus 0.375\baselineskip minus 0.1875\baselineskip +\else + \ifCLASSOPTIONconference% conference + \vskip 0.6\baselineskip + \else% + \ifCLASSOPTIONtechnote% technote + \vskip 1\baselineskip plus 0.375\baselineskip minus 0.1875\baselineskip% + \else% journal uses more space + \vskip 2.5\baselineskip plus 0.75\baselineskip minus 0.375\baselineskip% + \fi + \fi +\fi}} + + +% V1.6 +% This is a dynamically determined rigid spacer between the title/authors +% and the main text. This is used only for single column titles over two +% column text (most common) +% This is bit tricky because we have to ensure that the textheight of the +% main text is an integer multiple of \baselineskip +% otherwise underfull vbox problems may develop in the second column of the +% text on the titlepage +% The possible use of \IEEEpubid must also be taken into account. +\def\@IEEEdynamictitlevspace{{% + % we run within a group so that all the macros can be forgotten when we are done + \long\def\thanks##1{\relax}%don't allow \thanks to run when we evaluate the vbox height + \long\def\IEEEcompsocitemizethanks##1{\relax}%don't allow \IEEEcompsocitemizethanks to run when we evaluate the vbox height + \normalfont\normalsize% we declare more descriptive variable names + \let\@IEEEmaintextheight=\@IEEEtrantmpdimenA%height of the main text columns + \let\@IEEEINTmaintextheight=\@IEEEtrantmpdimenB%height of the main text columns with integer # lines + % set the nominal and minimum values for the title spacer + % the dynamic algorithm will not allow the spacer size to + % become less than \@IEEEMINtitlevspace - instead it will be + % lengthened + % default to journal values + \def\@IEEENORMtitlevspace{2.5\baselineskip}% + \def\@IEEEMINtitlevspace{2\baselineskip}% + % conferences and technotes need tighter spacing + \ifCLASSOPTIONconference%conference + \def\@IEEENORMtitlevspace{1\baselineskip}% + \def\@IEEEMINtitlevspace{0.75\baselineskip}% + \fi + \ifCLASSOPTIONtechnote%technote + \def\@IEEENORMtitlevspace{1\baselineskip}% + \def\@IEEEMINtitlevspace{0.75\baselineskip}% + \fi% + % get the height that the title will take up + \ifCLASSOPTIONpeerreview + \settoheight{\@IEEEmaintextheight}{\vbox{\hsize\textwidth \@IEEEpeerreviewmaketitle}}% + \else + \settoheight{\@IEEEmaintextheight}{\vbox{\hsize\textwidth \@maketitle}}% + \fi + \@IEEEmaintextheight=-\@IEEEmaintextheight% title takes away from maintext, so reverse sign + % add the height of the page textheight + \advance\@IEEEmaintextheight by \textheight% + % correct for title pages using pubid + \ifCLASSOPTIONpeerreview\else + % peerreview papers use the pubid on the cover page only. + % And the cover page uses a static spacer. + \if@IEEEusingpubid\advance\@IEEEmaintextheight by -\@IEEEpubidpullup\fi + \fi% + % subtract off the nominal value of the title bottom spacer + \advance\@IEEEmaintextheight by -\@IEEENORMtitlevspace% + % \topskip takes away some too + \advance\@IEEEmaintextheight by -\topskip% + % calculate the column height of the main text for lines + % now we calculate the main text height as if holding + % an integer number of \normalsize lines after the first + % and discard any excess fractional remainder + % we subtracted the first line, because the first line + % is placed \topskip into the maintext, not \baselineskip like the + % rest of the lines. + \@IEEEINTmaintextheight=\@IEEEmaintextheight% + \divide\@IEEEINTmaintextheight by \baselineskip% + \multiply\@IEEEINTmaintextheight by \baselineskip% + % now we calculate how much the title spacer height will + % have to be reduced from nominal (\@IEEEREDUCEmaintextheight is always + % a positive value) so that the maintext area will contain an integer + % number of normal size lines + % we change variable names here (to avoid confusion) as we no longer + % need \@IEEEINTmaintextheight and can reuse its dimen register + \let\@IEEEREDUCEmaintextheight=\@IEEEINTmaintextheight% + \advance\@IEEEREDUCEmaintextheight by -\@IEEEmaintextheight% + \advance\@IEEEREDUCEmaintextheight by \baselineskip% + % this is the calculated height of the spacer + % we change variable names here (to avoid confusion) as we no longer + % need \@IEEEmaintextheight and can reuse its dimen register + \let\@IEEECOMPENSATElen=\@IEEEmaintextheight% + \@IEEECOMPENSATElen=\@IEEENORMtitlevspace% set the nominal value + % we go with the reduced length if it is smaller than an increase + \ifdim\@IEEEREDUCEmaintextheight < 0.5\baselineskip\relax% + \advance\@IEEECOMPENSATElen by -\@IEEEREDUCEmaintextheight% + % if the resulting spacer is too small back out and go with an increase instead + \ifdim\@IEEECOMPENSATElen<\@IEEEMINtitlevspace\relax% + \advance\@IEEECOMPENSATElen by \baselineskip% + \fi% + \else% + % go with an increase because it is closer to the nominal than a decrease + \advance\@IEEECOMPENSATElen by -\@IEEEREDUCEmaintextheight% + \advance\@IEEECOMPENSATElen by \baselineskip% + \fi% + % set the calculated rigid spacer + \vspace{\@IEEECOMPENSATElen}}} + + + +% V1.6 +% we allow the user access to the last part of the title area +% useful in emergencies such as when a different spacing is needed +% This text is NOT compensated for in the dynamic sizer. +\let\@IEEEaftertitletext=\relax +\long\def\IEEEaftertitletext#1{\def\@IEEEaftertitletext{#1}} + +% V1.7 provide a way for users to enter abstract and keywords +% into the onecolumn title are. This text is compensated for +% in the dynamic sizer. +\let\@IEEEcompsoctitleabstractindextext=\relax +\long\def\IEEEcompsoctitleabstractindextext#1{\def\@IEEEcompsoctitleabstractindextext{#1}} +% V1.7 provide a way for users to get the \@IEEEcompsoctitleabstractindextext if +% not in compsoc journal mode - this way abstract and keywords can be placed +% in their conventional position if not in compsoc mode. +\def\IEEEdisplaynotcompsoctitleabstractindextext{% +\ifCLASSOPTIONcompsoc% display if compsoc conf +\ifCLASSOPTIONconference\@IEEEcompsoctitleabstractindextext\fi +\else% or if not compsoc +\@IEEEcompsoctitleabstractindextext\fi} + + +% command to allow alteration of baselinestretch, but only if the current +% baselineskip is unity. Used to tweak the compsoc abstract and keywords line spacing. +\def\@IEEEtweakunitybaselinestretch#1{{\def\baselinestretch{1}\selectfont +\global\@tempskipa\baselineskip}\ifnum\@tempskipa=\baselineskip% +\def\baselinestretch{#1}\selectfont\fi\relax} + + +% abstract and keywords are in \small, except +% for 9pt docs in which they are in \footnotesize +% Because 9pt docs use an 8pt footnotesize, \small +% becomes a rather awkward 8.5pt +\def\@IEEEabskeysecsize{\small} +\ifx\CLASSOPTIONpt\@IEEEptsizenine + \def\@IEEEabskeysecsize{\footnotesize} +\fi + +% compsoc journals use \footnotesize, compsoc conferences use normalsize +\@IEEEcompsoconly{\def\@IEEEabskeysecsize{\footnotesize}} +\@IEEEcompsocconfonly{\def\@IEEEabskeysecsize{\normalsize}} + + + + +% V1.6 have abstract and keywords strip leading spaces, pars and newlines +% so that spacing is more tightly controlled. +\def\abstract{\normalfont + \if@twocolumn + \par\@IEEEabskeysecsize\bfseries\leavevmode\kern-1pt\textit{\abstractname}---\relax + \else + \begin{center}\vspace{-1.78ex}\@IEEEabskeysecsize\textbf{\abstractname}\end{center}\quotation\@IEEEabskeysecsize + \fi\@IEEEgobbleleadPARNLSP} +% V1.6 IEEE wants only 1 pica from end of abstract to introduction heading when in +% conference mode (the heading already has this much above it) +\def\endabstract{\relax\ifCLASSOPTIONconference\vspace{0ex}\else\vspace{1.34ex}\fi\par\if@twocolumn\else\endquotation\fi + \normalfont\normalsize} + +\def\IEEEkeywords{\normalfont + \if@twocolumn + \@IEEEabskeysecsize\bfseries\leavevmode\kern-1pt\textit{\IEEEkeywordsname}---\relax + \else + \begin{center}\@IEEEabskeysecsize\textbf{\IEEEkeywordsname}\end{center}\quotation\@IEEEabskeysecsize + \fi\itshape\@IEEEgobbleleadPARNLSP} +\def\endIEEEkeywords{\relax\ifCLASSOPTIONtechnote\vspace{1.34ex}\else\vspace{0.5ex}\fi + \par\if@twocolumn\else\endquotation\fi% + \normalfont\normalsize} + +% V1.7 compsoc keywords index terms +\ifCLASSOPTIONcompsoc + \ifCLASSOPTIONconference% compsoc conference +\def\abstract{\normalfont + \begin{center}\@IEEEabskeysecsize\textbf{\large\abstractname}\end{center}\vskip 0.5\baselineskip plus 0.1\baselineskip minus 0.1\baselineskip + \if@twocolumn\else\quotation\fi\itshape\@IEEEabskeysecsize% + \par\@IEEEgobbleleadPARNLSP} +\def\IEEEkeywords{\normalfont\vskip 1.5\baselineskip plus 0.25\baselineskip minus 0.25\baselineskip + \begin{center}\@IEEEabskeysecsize\textbf{\large\IEEEkeywordsname}\end{center}\vskip 0.5\baselineskip plus 0.1\baselineskip minus 0.1\baselineskip + \if@twocolumn\else\quotation\fi\itshape\@IEEEabskeysecsize% + \par\@IEEEgobbleleadPARNLSP} + \else% compsoc not conference +\def\abstract{\normalfont\@IEEEtweakunitybaselinestretch{1.15}\sffamily + \if@twocolumn + \@IEEEabskeysecsize\noindent\textbf{\abstractname}---\relax + \else + \begin{center}\vspace{-1.78ex}\@IEEEabskeysecsize\textbf{\abstractname}\end{center}\quotation\@IEEEabskeysecsize% + \fi\@IEEEgobbleleadPARNLSP} +\def\IEEEkeywords{\normalfont\@IEEEtweakunitybaselinestretch{1.15}\sffamily + \if@twocolumn + \@IEEEabskeysecsize\vskip 0.5\baselineskip plus 0.25\baselineskip minus 0.25\baselineskip\noindent + \textbf{\IEEEkeywordsname}---\relax + \else + \begin{center}\@IEEEabskeysecsize\textbf{\IEEEkeywordsname}\end{center}\quotation\@IEEEabskeysecsize% + \fi\@IEEEgobbleleadPARNLSP} + \fi +\fi + + + +% gobbles all leading \, \\ and \par, upon finding first token that +% is not a \ , \\ or a \par, it ceases and returns that token +% +% used to strip leading \, \\ and \par from the input +% so that such things in the beginning of an environment will not +% affect the formatting of the text +\long\def\@IEEEgobbleleadPARNLSP#1{\let\@IEEEswallowthistoken=0% +\let\@IEEEgobbleleadPARNLSPtoken#1% +\let\@IEEEgobbleleadPARtoken=\par% +\let\@IEEEgobbleleadNLtoken=\\% +\let\@IEEEgobbleleadSPtoken=\ % +\def\@IEEEgobbleleadSPMACRO{\ }% +\ifx\@IEEEgobbleleadPARNLSPtoken\@IEEEgobbleleadPARtoken% +\let\@IEEEswallowthistoken=1% +\fi% +\ifx\@IEEEgobbleleadPARNLSPtoken\@IEEEgobbleleadNLtoken% +\let\@IEEEswallowthistoken=1% +\fi% +\ifx\@IEEEgobbleleadPARNLSPtoken\@IEEEgobbleleadSPtoken% +\let\@IEEEswallowthistoken=1% +\fi% +% a control space will come in as a macro +% when it is the last one on a line +\ifx\@IEEEgobbleleadPARNLSPtoken\@IEEEgobbleleadSPMACRO% +\let\@IEEEswallowthistoken=1% +\fi% +% if we have to swallow this token, do so and taste the next one +% else spit it out and stop gobbling +\ifx\@IEEEswallowthistoken 1\let\@IEEEnextgobbleleadPARNLSP=\@IEEEgobbleleadPARNLSP\else% +\let\@IEEEnextgobbleleadPARNLSP=#1\fi% +\@IEEEnextgobbleleadPARNLSP}% + + + + +% TITLING OF SECTIONS +\def\@IEEEsectpunct{:\ \,} % Punctuation after run-in section heading (headings which are + % part of the paragraphs), need little bit more than a single space + % spacing from section number to title +% compsoc conferences use regular period/space punctuation +\ifCLASSOPTIONcompsoc +\ifCLASSOPTIONconference +\def\@IEEEsectpunct{.\ } +\fi\fi + +\def\@seccntformat#1{\hb@xt@ 1.4em{\csname the#1dis\endcsname\hss\relax}} +\def\@seccntformatinl#1{\hb@xt@ 1.1em{\csname the#1dis\endcsname\hss\relax}} +\def\@seccntformatch#1{\csname the#1dis\endcsname\hskip 1em\relax} + +\ifCLASSOPTIONcompsoc +% compsoc journals need extra spacing +\ifCLASSOPTIONconference\else +\def\@seccntformat#1{\csname the#1dis\endcsname\hskip 1em\relax} +\fi\fi + +%v1.7 put {} after #6 to allow for some types of user font control +%and use \@@par rather than \par +\def\@sect#1#2#3#4#5#6[#7]#8{% + \ifnum #2>\c@secnumdepth + \let\@svsec\@empty + \else + \refstepcounter{#1}% + % load section label and spacer into \@svsec + \ifnum #2=1 + \protected@edef\@svsec{\@seccntformatch{#1}\relax}% + \else + \ifnum #2>2 + \protected@edef\@svsec{\@seccntformatinl{#1}\relax}% + \else + \protected@edef\@svsec{\@seccntformat{#1}\relax}% + \fi + \fi + \fi% + \@tempskipa #5\relax + \ifdim \@tempskipa>\z@% tempskipa determines whether is treated as a high + \begingroup #6{\relax% or low level heading + \noindent % subsections are NOT indented + % print top level headings. \@svsec is label, #8 is heading title + % IEEE does not block indent the section title text, it flows like normal + {\hskip #3\relax\@svsec}{\interlinepenalty \@M #8\@@par}}% + \endgroup + \addcontentsline{toc}{#1}{\ifnum #2>\c@secnumdepth\relax\else + \protect\numberline{\csname the#1\endcsname}\fi#7}% + \else % printout low level headings + % svsechd seems to swallow the trailing space, protect it with \mbox{} + % got rid of sectionmark stuff + \def\@svsechd{#6{\hskip #3\relax\@svsec #8\@IEEEsectpunct\mbox{}}% + \addcontentsline{toc}{#1}{\ifnum #2>\c@secnumdepth\relax\else + \protect\numberline{\csname the#1\endcsname}\fi#7}}% + \fi%skip down + \@xsect{#5}} + + +% section* handler +%v1.7 put {} after #4 to allow for some types of user font control +%and use \@@par rather than \par +\def\@ssect#1#2#3#4#5{\@tempskipa #3\relax + \ifdim \@tempskipa>\z@ + %\begingroup #4\@hangfrom{\hskip #1}{\interlinepenalty \@M #5\par}\endgroup + % IEEE does not block indent the section title text, it flows like normal + \begingroup \noindent #4{\relax{\hskip #1}{\interlinepenalty \@M #5\@@par}}\endgroup + % svsechd swallows the trailing space, protect it with \mbox{} + \else \def\@svsechd{#4{\hskip #1\relax #5\@IEEEsectpunct\mbox{}}}\fi + \@xsect{#3}} + + +%% SECTION heading spacing and font +%% +% arguments are: #1 - sectiontype name +% (for \@sect) #2 - section level +% #3 - section heading indent +% #4 - top separation (absolute value used, neg indicates not to indent main text) +% If negative, make stretch parts negative too! +% #5 - (absolute value used) positive: bottom separation after heading, +% negative: amount to indent main text after heading +% Both #4 and #5 negative means to indent main text and use negative top separation +% #6 - font control +% You've got to have \normalfont\normalsize in the font specs below to prevent +% trouble when you do something like: +% \section{Note}{\ttfamily TT-TEXT} is known to ... +% IEEE sometimes REALLY stretches the area before a section +% heading by up to about 0.5in. However, it may not be a good +% idea to let LaTeX have quite this much rubber. +\ifCLASSOPTIONconference% +% IEEE wants section heading spacing to decrease for conference mode +\def\section{\@startsection{section}{1}{\z@}{1.5ex plus 1.5ex minus 0.5ex}% +{1sp}{\normalfont\normalsize\centering\scshape}}% +\def\subsection{\@startsection{subsection}{2}{\z@}{1.5ex plus 1.5ex minus 0.5ex}% +{1sp}{\normalfont\normalsize\itshape}}% +\else % for journals +\def\section{\@startsection{section}{1}{\z@}{3.0ex plus 1.5ex minus 1.5ex}% V1.6 3.0ex from 3.5ex +{0.7ex plus 1ex minus 0ex}{\normalfont\normalsize\centering\scshape}}% +\def\subsection{\@startsection{subsection}{2}{\z@}{3.5ex plus 1.5ex minus 1.5ex}% +{0.7ex plus .5ex minus 0ex}{\normalfont\normalsize\itshape}}% +\fi + +% for both journals and conferences +% decided to put in a little rubber above the section, might help somebody +\def\subsubsection{\@startsection{subsubsection}{3}{\parindent}{0ex plus 0.1ex minus 0.1ex}% +{0ex}{\normalfont\normalsize\itshape}}% +\def\paragraph{\@startsection{paragraph}{4}{2\parindent}{0ex plus 0.1ex minus 0.1ex}% +{0ex}{\normalfont\normalsize\itshape}}% + + +% compsoc +\ifCLASSOPTIONcompsoc +\ifCLASSOPTIONconference +% compsoc conference +\def\section{\@startsection{section}{1}{\z@}{1\baselineskip plus 0.25\baselineskip minus 0.25\baselineskip}% +{1\baselineskip plus 0.25\baselineskip minus 0.25\baselineskip}{\normalfont\large\bfseries}}% +\def\subsection{\@startsection{subsection}{2}{\z@}{1\baselineskip plus 0.25\baselineskip minus 0.25\baselineskip}% +{1\baselineskip plus 0.25\baselineskip minus 0.25\baselineskip}{\normalfont\sublargesize\bfseries}}% +\def\subsubsection{\@startsection{subsubsection}{3}{\z@}{1\baselineskip plus 0.25\baselineskip minus 0.25\baselineskip}% +{0ex}{\normalfont\normalsize\bfseries}}% +\def\paragraph{\@startsection{paragraph}{4}{2\parindent}{0ex plus 0.1ex minus 0.1ex}% +{0ex}{\normalfont\normalsize}}% +\else% compsoc journals +% use negative top separation as compsoc journals do not indent paragraphs after section titles +\def\section{\@startsection{section}{1}{\z@}{-3ex plus -2ex minus -1.5ex}% +{0.7ex plus 1ex minus 0ex}{\normalfont\large\sffamily\bfseries\scshape}}% +% Note that subsection and smaller may not be correct for the Computer Society, +% I have to look up an example. +\def\subsection{\@startsection{subsection}{2}{\z@}{-3.5ex plus -1.5ex minus -1.5ex}% +{0.7ex plus .5ex minus 0ex}{\normalfont\normalsize\sffamily\bfseries}}% +\def\subsubsection{\@startsection{subsubsection}{3}{\z@}{-2.5ex plus -1ex minus -1ex}% +{0.5ex plus 0.5ex minus 0ex}{\normalfont\normalsize\sffamily\itshape}}% +\def\paragraph{\@startsection{paragraph}{4}{2\parindent}{-0ex plus -0.1ex minus -0.1ex}% +{0ex}{\normalfont\normalsize}}% +\fi\fi + + + + +%% ENVIRONMENTS +% "box" symbols at end of proofs +\def\IEEEQEDclosed{\mbox{\rule[0pt]{1.3ex}{1.3ex}}} % for a filled box +% V1.6 some journals use an open box instead that will just fit around a closed one +\def\IEEEQEDopen{{\setlength{\fboxsep}{0pt}\setlength{\fboxrule}{0.2pt}\fbox{\rule[0pt]{0pt}{1.3ex}\rule[0pt]{1.3ex}{0pt}}}} +\ifCLASSOPTIONcompsoc +\def\IEEEQED{\IEEEQEDopen} % default to open for compsoc +\else +\def\IEEEQED{\IEEEQEDclosed} % otherwise default to closed +\fi + +% v1.7 name change to avoid namespace collision with amsthm. Also add support +% for an optional argument. +\def\IEEEproof{\@ifnextchar[{\@IEEEproof}{\@IEEEproof[\IEEEproofname]}} +\def\@IEEEproof[#1]{\par\noindent\hspace{2em}{\itshape #1: }} +\def\endIEEEproof{\hspace*{\fill}~\IEEEQED\par} + + +%\itemindent is set to \z@ by list, so define new temporary variable +\newdimen\@IEEEtmpitemindent +\def\@begintheorem#1#2{\@IEEEtmpitemindent\itemindent\topsep 0pt\rmfamily\trivlist% + \item[\hskip \labelsep{\indent\itshape #1\ #2:}]\itemindent\@IEEEtmpitemindent} +\def\@opargbegintheorem#1#2#3{\@IEEEtmpitemindent\itemindent\topsep 0pt\rmfamily \trivlist% +% V1.6 IEEE is back to using () around theorem names which are also in italics +% Thanks to Christian Peel for reporting this. + \item[\hskip\labelsep{\indent\itshape #1\ #2\ (#3):}]\itemindent\@IEEEtmpitemindent} +% V1.7 remove bogus \unskip that caused equations in theorems to collide with +% lines below. +\def\@endtheorem{\endtrivlist} + +% V1.6 +% display command for the section the theorem is in - so that \thesection +% is not used as this will be in Roman numerals when we want arabic. +% LaTeX2e uses \def\@thmcounter#1{\noexpand\arabic{#1}} for the theorem number +% (second part) display and \def\@thmcountersep{.} as a separator. +% V1.7 intercept calls to the section counter and reroute to \@IEEEthmcounterinsection +% to allow \appendix(ices} to override as needed. +% +% special handler for sections, allows appendix(ices) to override +\gdef\@IEEEthmcounterinsection#1{\arabic{#1}} +% string macro +\edef\@IEEEstringsection{section} + +% redefine the #1#2[#3] form of newtheorem to use a hook to \@IEEEthmcounterinsection +% if section in_counter is used +\def\@xnthm#1#2[#3]{% + \expandafter\@ifdefinable\csname #1\endcsname + {\@definecounter{#1}\@newctr{#1}[#3]% + \edef\@IEEEstringtmp{#3} + \ifx\@IEEEstringtmp\@IEEEstringsection + \expandafter\xdef\csname the#1\endcsname{% + \noexpand\@IEEEthmcounterinsection{#3}\@thmcountersep + \@thmcounter{#1}}% + \else + \expandafter\xdef\csname the#1\endcsname{% + \expandafter\noexpand\csname the#3\endcsname \@thmcountersep + \@thmcounter{#1}}% + \fi + \global\@namedef{#1}{\@thm{#1}{#2}}% + \global\@namedef{end#1}{\@endtheorem}}} + + + +%% SET UP THE DEFAULT PAGESTYLE +\ps@headings +\pagenumbering{arabic} + +% normally the page counter starts at 1 +\setcounter{page}{1} +% however, for peerreview the cover sheet is page 0 or page -1 +% (for duplex printing) +\ifCLASSOPTIONpeerreview + \if@twoside + \setcounter{page}{-1} + \else + \setcounter{page}{0} + \fi +\fi + +% standard book class behavior - let bottom line float up and down as +% needed when single sided +\ifCLASSOPTIONtwoside\else\raggedbottom\fi +% if two column - turn on twocolumn, allow word spacings to stretch more and +% enforce a rigid position for the last lines +\ifCLASSOPTIONtwocolumn +% the peer review option delays invoking twocolumn + \ifCLASSOPTIONpeerreview\else + \twocolumn + \fi +\sloppy +\flushbottom +\fi + + + + +% \APPENDIX and \APPENDICES definitions + +% This is the \@ifmtarg command from the LaTeX ifmtarg package +% by Peter Wilson (CUA) and Donald Arseneau +% \@ifmtarg is used to determine if an argument to a command +% is present or not. +% For instance: +% \@ifmtarg{#1}{\typeout{empty}}{\typeout{has something}} +% \@ifmtarg is used with our redefined \section command if +% \appendices is invoked. +% The command \section will behave slightly differently depending +% on whether the user specifies a title: +% \section{My appendix title} +% or not: +% \section{} +% This way, we can eliminate the blank lines where the title +% would be, and the unneeded : after Appendix in the table of +% contents +\begingroup +\catcode`\Q=3 +\long\gdef\@ifmtarg#1{\@xifmtarg#1QQ\@secondoftwo\@firstoftwo\@nil} +\long\gdef\@xifmtarg#1#2Q#3#4#5\@nil{#4} +\endgroup +% end of \@ifmtarg defs + + +% V1.7 +% command that allows the one time saving of the original definition +% of section to \@IEEEappendixsavesection for \appendix or \appendices +% we don't save \section here as it may be redefined later by other +% packages (hyperref.sty, etc.) +\def\@IEEEsaveoriginalsectiononce{\let\@IEEEappendixsavesection\section +\let\@IEEEsaveoriginalsectiononce\relax} + +% neat trick to grab and process the argument from \section{argument} +% we process differently if the user invoked \section{} with no +% argument (title) +% note we reroute the call to the old \section* +\def\@IEEEprocessthesectionargument#1{% +\@ifmtarg{#1}{% +\@IEEEappendixsavesection*{\appendixname~\thesectiondis}% +\addcontentsline{toc}{section}{\appendixname~\thesection}}{% +\@IEEEappendixsavesection*{\appendixname~\thesectiondis \\* #1}% +\addcontentsline{toc}{section}{\appendixname~\thesection: #1}}} + +% we use this if the user calls \section{} after +% \appendix-- which has no meaning. So, we ignore the +% command and its argument. Then, warn the user. +\def\@IEEEdestroythesectionargument#1{\typeout{** WARNING: Ignoring useless +\protect\section\space in Appendix (line \the\inputlineno).}} + + +% remember \thesection forms will be displayed in \ref calls +% and in the Table of Contents. +% The \sectiondis form is used in the actual heading itself + +% appendix command for one single appendix +% normally has no heading. However, if you want a +% heading, you can do so via the optional argument: +% \appendix[Optional Heading] +\def\appendix{\relax} +\renewcommand{\appendix}[1][]{\@IEEEsaveoriginalsectiononce\par + % v1.6 keep hyperref's identifiers unique + \gdef\theHsection{Appendix.A}% + % v1.6 adjust hyperref's string name for the section + \xdef\Hy@chapapp{appendix}% + \setcounter{section}{0}% + \setcounter{subsection}{0}% + \setcounter{subsubsection}{0}% + \setcounter{paragraph}{0}% + \gdef\thesection{A}% + \gdef\thesectiondis{}% + \gdef\thesubsection{\Alph{subsection}}% + \gdef\@IEEEthmcounterinsection##1{A} + \refstepcounter{section}% update the \ref counter + \@ifmtarg{#1}{\@IEEEappendixsavesection*{\appendixname}% + \addcontentsline{toc}{section}{\appendixname}}{% + \@IEEEappendixsavesection*{\appendixname~\\* #1}% + \addcontentsline{toc}{section}{\appendixname: #1}}% + % redefine \section command for appendix + % leave \section* as is + \def\section{\@ifstar{\@IEEEappendixsavesection*}{% + \@IEEEdestroythesectionargument}}% throw out the argument + % of the normal form +} + + + +% appendices command for multiple appendices +% user then calls \section with an argument (possibly empty) to +% declare the individual appendices +\def\appendices{\@IEEEsaveoriginalsectiononce\par + % v1.6 keep hyperref's identifiers unique + \gdef\theHsection{Appendix.\Alph{section}}% + % v1.6 adjust hyperref's string name for the section + \xdef\Hy@chapapp{appendix}% + \setcounter{section}{-1}% we want \refstepcounter to use section 0 + \setcounter{subsection}{0}% + \setcounter{subsubsection}{0}% + \setcounter{paragraph}{0}% + \ifCLASSOPTIONromanappendices% + \gdef\thesection{\Roman{section}}% + \gdef\thesectiondis{\Roman{section}}% + \@IEEEcompsocconfonly{\gdef\thesectiondis{\Roman{section}.}}% + \gdef\@IEEEthmcounterinsection##1{A\arabic{##1}} + \else% + \gdef\thesection{\Alph{section}}% + \gdef\thesectiondis{\Alph{section}}% + \@IEEEcompsocconfonly{\gdef\thesectiondis{\Alph{section}.}}% + \gdef\@IEEEthmcounterinsection##1{\Alph{##1}} + \fi% + \refstepcounter{section}% update the \ref counter + \setcounter{section}{0}% NEXT \section will be the FIRST appendix + % redefine \section command for appendices + % leave \section* as is + \def\section{\@ifstar{\@IEEEappendixsavesection*}{% process the *-form + \refstepcounter{section}% or is a new section so, + \@IEEEprocessthesectionargument}}% process the argument + % of the normal form +} + + + +% \IEEEPARstart +% Definition for the big two line drop cap letter at the beginning of the +% first paragraph of journal papers. The first argument is the first letter +% of the first word, the second argument is the remaining letters of the +% first word which will be rendered in upper case. +% In V1.6 this has been completely rewritten to: +% +% 1. no longer have problems when the user begins an environment +% within the paragraph that uses \IEEEPARstart. +% 2. auto-detect and use the current font family +% 3. revise handling of the space at the end of the first word so that +% interword glue will now work as normal. +% 4. produce correctly aligned edges for the (two) indented lines. +% +% We generalize things via control macros - playing with these is fun too. +% +% V1.7 added more control macros to make it easy for IEEEtrantools.sty users +% to change the font style. +% +% the number of lines that are indented to clear it +% may need to increase if using decenders +\def\@IEEEPARstartDROPLINES{2} +% minimum number of lines left on a page to allow a \@IEEEPARstart +% Does not take into consideration rubber shrink, so it tends to +% be overly cautious +\def\@IEEEPARstartMINPAGELINES{2} +% V1.7 the height of the drop cap is adjusted to match the height of this text +% in the current font (when \IEEEPARstart is called). +\def\@IEEEPARstartHEIGHTTEXT{T} +% the depth the letter is lowered below the baseline +% the height (and size) of the letter is determined by the sum +% of this value and the height of the \@IEEEPARstartHEIGHTTEXT in the current +% font. It is a good idea to set this value in terms of the baselineskip +% so that it can respond to changes therein. +\def\@IEEEPARstartDROPDEPTH{1.1\baselineskip} +% V1.7 the font the drop cap will be rendered in, +% can take zero or one argument. +\def\@IEEEPARstartFONTSTYLE{\bfseries} +% V1.7 any additional, non-font related commands needed to modify +% the drop cap letter, can take zero or one argument. +\def\@IEEEPARstartCAPSTYLE{\MakeUppercase} +% V1.7 the font that will be used to render the rest of the word, +% can take zero or one argument. +\def\@IEEEPARstartWORDFONTSTYLE{\relax} +% V1.7 any additional, non-font related commands needed to modify +% the rest of the word, can take zero or one argument. +\def\@IEEEPARstartWORDCAPSTYLE{\MakeUppercase} +% This is the horizontal separation distance from the drop letter to the main text. +% Lengths that depend on the font (e.g., ex, em, etc.) will be referenced +% to the font that is active when \IEEEPARstart is called. +\def\@IEEEPARstartSEP{0.15em} +% V1.7 horizontal offset applied to the left of the drop cap. +\def\@IEEEPARstartHOFFSET{0em} +% V1.7 Italic correction command applied at the end of the drop cap. +\def\@IEEEPARstartITLCORRECT{\/} + +% V1.7 compoc uses nonbold drop cap and small caps word style +\ifCLASSOPTIONcompsoc +\def\@IEEEPARstartFONTSTYLE{\mdseries} +\def\@IEEEPARstartWORDFONTSTYLE{\scshape} +\def\@IEEEPARstartWORDCAPSTYLE{\relax} +\fi + +% definition of \IEEEPARstart +% THIS IS A CONTROLLED SPACING AREA, DO NOT ALLOW SPACES WITHIN THESE LINES +% +% The token \@IEEEPARstartfont will be globally defined after the first use +% of \IEEEPARstart and will be a font command which creates the big letter +% The first argument is the first letter of the first word and the second +% argument is the rest of the first word(s). +\def\IEEEPARstart#1#2{\par{% +% if this page does not have enough space, break it and lets start +% on a new one +\@IEEEtranneedspace{\@IEEEPARstartMINPAGELINES\baselineskip}{\relax}% +% V1.7 move this up here in case user uses \textbf for \@IEEEPARstartFONTSTYLE +% which uses command \leavevmode which causes an unwanted \indent to be issued +\noindent +% calculate the desired height of the big letter +% it extends from the top of \@IEEEPARstartHEIGHTTEXT in the current font +% down to \@IEEEPARstartDROPDEPTH below the current baseline +\settoheight{\@IEEEtrantmpdimenA}{\@IEEEPARstartHEIGHTTEXT}% +\addtolength{\@IEEEtrantmpdimenA}{\@IEEEPARstartDROPDEPTH}% +% extract the name of the current font in bold +% and place it in \@IEEEPARstartFONTNAME +\def\@IEEEPARstartGETFIRSTWORD##1 ##2\relax{##1}% +{\@IEEEPARstartFONTSTYLE{\selectfont\edef\@IEEEPARstartFONTNAMESPACE{\fontname\font\space}% +\xdef\@IEEEPARstartFONTNAME{\expandafter\@IEEEPARstartGETFIRSTWORD\@IEEEPARstartFONTNAMESPACE\relax}}}% +% define a font based on this name with a point size equal to the desired +% height of the drop letter +\font\@IEEEPARstartsubfont\@IEEEPARstartFONTNAME\space at \@IEEEtrantmpdimenA\relax% +% save this value as a counter (integer) value (sp points) +\@IEEEtrantmpcountA=\@IEEEtrantmpdimenA% +% now get the height of the actual letter produced by this font size +\settoheight{\@IEEEtrantmpdimenB}{\@IEEEPARstartsubfont\@IEEEPARstartCAPSTYLE{#1}}% +% If something bogus happens like the first argument is empty or the +% current font is strange, do not allow a zero height. +\ifdim\@IEEEtrantmpdimenB=0pt\relax% +\typeout{** WARNING: IEEEPARstart drop letter has zero height! (line \the\inputlineno)}% +\typeout{ Forcing the drop letter font size to 10pt.}% +\@IEEEtrantmpdimenB=10pt% +\fi% +% and store it as a counter +\@IEEEtrantmpcountB=\@IEEEtrantmpdimenB% +% Since a font size doesn't exactly correspond to the height of the capital +% letters in that font, the actual height of the letter, \@IEEEtrantmpcountB, +% will be less than that desired, \@IEEEtrantmpcountA +% we need to raise the font size, \@IEEEtrantmpdimenA +% by \@IEEEtrantmpcountA / \@IEEEtrantmpcountB +% But, TeX doesn't have floating point division, so we have to use integer +% division. Hence the use of the counters. +% We need to reduce the denominator so that the loss of the remainder will +% have minimal affect on the accuracy of the result +\divide\@IEEEtrantmpcountB by 200% +\divide\@IEEEtrantmpcountA by \@IEEEtrantmpcountB% +% Then reequalize things when we use TeX's ability to multiply by +% floating point values +\@IEEEtrantmpdimenB=0.005\@IEEEtrantmpdimenA% +\multiply\@IEEEtrantmpdimenB by \@IEEEtrantmpcountA% +% \@IEEEPARstartfont is globaly set to the calculated font of the big letter +% We need to carry this out of the local calculation area to to create the +% big letter. +\global\font\@IEEEPARstartfont\@IEEEPARstartFONTNAME\space at \@IEEEtrantmpdimenB% +% Now set \@IEEEtrantmpdimenA to the width of the big letter +% We need to carry this out of the local calculation area to set the +% hanging indent +\settowidth{\global\@IEEEtrantmpdimenA}{\@IEEEPARstartfont +\@IEEEPARstartCAPSTYLE{#1\@IEEEPARstartITLCORRECT}}}% +% end of the isolated calculation environment +% add in the extra clearance we want +\advance\@IEEEtrantmpdimenA by \@IEEEPARstartSEP\relax% +% add in the optional offset +\advance\@IEEEtrantmpdimenA by \@IEEEPARstartHOFFSET\relax% +% V1.7 don't allow negative offsets to produce negative hanging indents +\@IEEEtrantmpdimenB\@IEEEtrantmpdimenA +\ifnum\@IEEEtrantmpdimenB < 0 \@IEEEtrantmpdimenB 0pt\fi +% \@IEEEtrantmpdimenA has the width of the big letter plus the +% separation space and \@IEEEPARstartfont is the font we need to use +% Now, we make the letter and issue the hanging indent command +% The letter is placed in a box of zero width and height so that other +% text won't be displaced by it. +\hangindent\@IEEEtrantmpdimenB\hangafter=-\@IEEEPARstartDROPLINES% +\makebox[0pt][l]{\hspace{-\@IEEEtrantmpdimenA}% +\raisebox{-\@IEEEPARstartDROPDEPTH}[0pt][0pt]{\hspace{\@IEEEPARstartHOFFSET}% +\@IEEEPARstartfont\@IEEEPARstartCAPSTYLE{#1\@IEEEPARstartITLCORRECT}% +\hspace{\@IEEEPARstartSEP}}}% +{\@IEEEPARstartWORDFONTSTYLE{\@IEEEPARstartWORDCAPSTYLE{\selectfont#2}}}} + + + + + + +% determines if the space remaining on a given page is equal to or greater +% than the specified space of argument one +% if not, execute argument two (only if the remaining space is greater than zero) +% and issue a \newpage +% +% example: \@IEEEtranneedspace{2in}{\vfill} +% +% Does not take into consideration rubber shrinkage, so it tends to +% be overly cautious +% Based on an example posted by Donald Arseneau +% Note this macro uses \@IEEEtrantmpdimenB internally for calculations, +% so DO NOT PASS \@IEEEtrantmpdimenB to this routine +% if you need a dimen register, import with \@IEEEtrantmpdimenA instead +\def\@IEEEtranneedspace#1#2{\penalty-100\begingroup%shield temp variable +\@IEEEtrantmpdimenB\pagegoal\advance\@IEEEtrantmpdimenB-\pagetotal% space left +\ifdim #1>\@IEEEtrantmpdimenB\relax% not enough space left +\ifdim\@IEEEtrantmpdimenB>\z@\relax #2\fi% +\newpage% +\fi\endgroup} + + + +% IEEEbiography ENVIRONMENT +% Allows user to enter biography leaving place for picture (adapts to font size) +% As of V1.5, a new optional argument allows you to have a real graphic! +% V1.5 and later also fixes the "colliding biographies" which could happen when a +% biography's text was shorter than the space for the photo. +% MDS 7/2001 +% V1.6 prevent multiple biographies from making multiple TOC entries +\newif\if@IEEEbiographyTOCentrynotmade +\global\@IEEEbiographyTOCentrynotmadetrue + +% biography counter so hyperref can jump directly to the biographies +% and not just the previous section +\newcounter{IEEEbiography} +\setcounter{IEEEbiography}{0} + +% photo area size +\def\@IEEEBIOphotowidth{1.0in} % width of the biography photo area +\def\@IEEEBIOphotodepth{1.25in} % depth (height) of the biography photo area +% area cleared for photo +\def\@IEEEBIOhangwidth{1.14in} % width cleared for the biography photo area +\def\@IEEEBIOhangdepth{1.25in} % depth cleared for the biography photo area + % actual depth will be a multiple of + % \baselineskip, rounded up +\def\@IEEEBIOskipN{4\baselineskip}% nominal value of the vskip above the biography + +\newenvironment{IEEEbiography}[2][]{\normalfont\@IEEEcompsoconly{\sffamily}\footnotesize% +\unitlength 1in\parskip=0pt\par\parindent 1em\interlinepenalty500% +% we need enough space to support the hanging indent +% the nominal value of the spacer +% and one extra line for good measure +\@IEEEtrantmpdimenA=\@IEEEBIOhangdepth% +\advance\@IEEEtrantmpdimenA by \@IEEEBIOskipN% +\advance\@IEEEtrantmpdimenA by 1\baselineskip% +% if this page does not have enough space, break it and lets start +% with a new one +\@IEEEtranneedspace{\@IEEEtrantmpdimenA}{\relax}% +% nominal spacer can strech, not shrink use 1fil so user can out stretch with \vfill +\vskip \@IEEEBIOskipN plus 1fil minus 0\baselineskip% +% the default box for where the photo goes +\def\@IEEEtempbiographybox{{\setlength{\fboxsep}{0pt}\framebox{% +\begin{minipage}[b][\@IEEEBIOphotodepth][c]{\@IEEEBIOphotowidth}\centering PLACE\\ PHOTO\\ HERE \end{minipage}}}}% +% +% detect if the optional argument was supplied, this requires the +% \@ifmtarg command as defined in the appendix section above +% and if so, override the default box with what they want +\@ifmtarg{#1}{\relax}{\def\@IEEEtempbiographybox{\mbox{\begin{minipage}[b][\@IEEEBIOphotodepth][c]{\@IEEEBIOphotowidth}% +\centering% +#1% +\end{minipage}}}}% end if optional argument supplied +% Make an entry into the table of contents only if we have not done so before +\if@IEEEbiographyTOCentrynotmade% +% link labels to the biography counter so hyperref will jump +% to the biography, not the previous section +\setcounter{IEEEbiography}{-1}% +\refstepcounter{IEEEbiography}% +\addcontentsline{toc}{section}{Biographies}% +\global\@IEEEbiographyTOCentrynotmadefalse% +\fi% +% one more biography +\refstepcounter{IEEEbiography}% +% Make an entry for this name into the table of contents +\addcontentsline{toc}{subsection}{#2}% +% V1.6 properly handle if a new paragraph should occur while the +% hanging indent is still active. Do this by redefining \par so +% that it will not start a new paragraph. (But it will appear to the +% user as if it did.) Also, strip any leading pars, newlines, or spaces. +\let\@IEEEBIOORGparCMD=\par% save the original \par command +\edef\par{\hfil\break\indent}% the new \par will not be a "real" \par +\settoheight{\@IEEEtrantmpdimenA}{\@IEEEtempbiographybox}% get height of biography box +\@IEEEtrantmpdimenB=\@IEEEBIOhangdepth% +\@IEEEtrantmpcountA=\@IEEEtrantmpdimenB% countA has the hang depth +\divide\@IEEEtrantmpcountA by \baselineskip% calculates lines needed to produce the hang depth +\advance\@IEEEtrantmpcountA by 1% ensure we overestimate +% set the hanging indent +\hangindent\@IEEEBIOhangwidth% +\hangafter-\@IEEEtrantmpcountA% +% reference the top of the photo area to the top of a capital T +\settoheight{\@IEEEtrantmpdimenB}{\mbox{T}}% +% set the photo box, give it zero width and height so as not to disturb anything +\noindent\makebox[0pt][l]{\hspace{-\@IEEEBIOhangwidth}\raisebox{\@IEEEtrantmpdimenB}[0pt][0pt]{% +\raisebox{-\@IEEEBIOphotodepth}[0pt][0pt]{\@IEEEtempbiographybox}}}% +% now place the author name and begin the bio text +\noindent\textbf{#2\ }\@IEEEgobbleleadPARNLSP}{\relax\let\par=\@IEEEBIOORGparCMD\par% +% 7/2001 V1.5 detect when the biography text is shorter than the photo area +% and pad the unused area - preventing a collision from the next biography entry +% MDS +\ifnum \prevgraf <\@IEEEtrantmpcountA\relax% detect when the biography text is shorter than the photo + \advance\@IEEEtrantmpcountA by -\prevgraf% calculate how many lines we need to pad + \advance\@IEEEtrantmpcountA by -1\relax% we compensate for the fact that we indented an extra line + \@IEEEtrantmpdimenA=\baselineskip% calculate the length of the padding + \multiply\@IEEEtrantmpdimenA by \@IEEEtrantmpcountA% + \noindent\rule{0pt}{\@IEEEtrantmpdimenA}% insert an invisible support strut +\fi% +\par\normalfont} + + + +% V1.6 +% added biography without a photo environment +\newenvironment{IEEEbiographynophoto}[1]{% +% Make an entry into the table of contents only if we have not done so before +\if@IEEEbiographyTOCentrynotmade% +% link labels to the biography counter so hyperref will jump +% to the biography, not the previous section +\setcounter{IEEEbiography}{-1}% +\refstepcounter{IEEEbiography}% +\addcontentsline{toc}{section}{Biographies}% +\global\@IEEEbiographyTOCentrynotmadefalse% +\fi% +% one more biography +\refstepcounter{IEEEbiography}% +% Make an entry for this name into the table of contents +\addcontentsline{toc}{subsection}{#1}% +\normalfont\@IEEEcompsoconly{\sffamily}\footnotesize\interlinepenalty500% +\vskip 4\baselineskip plus 1fil minus 0\baselineskip% +\parskip=0pt\par% +\noindent\textbf{#1\ }\@IEEEgobbleleadPARNLSP}{\relax\par\normalfont} + + +% provide the user with some old font commands +% got this from article.cls +\DeclareOldFontCommand{\rm}{\normalfont\rmfamily}{\mathrm} +\DeclareOldFontCommand{\sf}{\normalfont\sffamily}{\mathsf} +\DeclareOldFontCommand{\tt}{\normalfont\ttfamily}{\mathtt} +\DeclareOldFontCommand{\bf}{\normalfont\bfseries}{\mathbf} +\DeclareOldFontCommand{\it}{\normalfont\itshape}{\mathit} +\DeclareOldFontCommand{\sl}{\normalfont\slshape}{\@nomath\sl} +\DeclareOldFontCommand{\sc}{\normalfont\scshape}{\@nomath\sc} +\DeclareRobustCommand*\cal{\@fontswitch\relax\mathcal} +\DeclareRobustCommand*\mit{\@fontswitch\relax\mathnormal} + + +% SPECIAL PAPER NOTICE COMMANDS +% +% holds the special notice text +\def\@IEEEspecialpapernotice{\relax} + +% for special papers, like invited papers, the user can do: +% \IEEEspecialpapernotice{(Invited Paper)} before \maketitle +\def\IEEEspecialpapernotice#1{\ifCLASSOPTIONconference% +\def\@IEEEspecialpapernotice{{\Large#1\vspace*{1em}}}% +\else% +\def\@IEEEspecialpapernotice{{\\*[1.5ex]\sublargesize\textit{#1}}\vspace*{-2ex}}% +\fi} + + + + +% PUBLISHER ID COMMANDS +% to insert a publisher's ID footer +% V1.6 \IEEEpubid has been changed so that the change in page size and style +% occurs in \maketitle. \IEEEpubid must now be issued prior to \maketitle +% use \IEEEpubidadjcol as before - in the second column of the title page +% These changes allow \maketitle to take the reduced page height into +% consideration when dynamically setting the space between the author +% names and the maintext. +% +% the amount the main text is pulled up to make room for the +% publisher's ID footer +% IEEE uses about 1.3\baselineskip for journals, +% dynamic title spacing will clean up the fraction +\def\@IEEEpubidpullup{1.3\baselineskip} +\ifCLASSOPTIONtechnote +% for technotes it must be an integer of baselineskip as there can be no +% dynamic title spacing for two column mode technotes (the title is in the +% in first column) and we should maintain an integer number of lines in the +% second column +% There are some examples (such as older issues of "Transactions on +% Information Theory") in which IEEE really pulls the text off the ID for +% technotes - about 0.55in (or 4\baselineskip). We'll use 2\baselineskip +% and call it even. +\def\@IEEEpubidpullup{2\baselineskip} +\fi + +% V1.7 compsoc does not use a pullup +\ifCLASSOPTIONcompsoc +\def\@IEEEpubidpullup{0pt} +\fi + +% holds the ID text +\def\@IEEEpubid{\relax} + +% flag so \maketitle can tell if \IEEEpubid was called +\newif\if@IEEEusingpubid +\global\@IEEEusingpubidfalse +% issue this command in the page to have the ID at the bottom +% V1.6 use before \maketitle +\def\IEEEpubid#1{\def\@IEEEpubid{#1}\global\@IEEEusingpubidtrue} + + +% command which will pull up (shorten) the column it is executed in +% to make room for the publisher ID. Place in the second column of +% the title page when using \IEEEpubid +% Is smart enough not to do anything when in single column text or +% if the user hasn't called \IEEEpubid +% currently needed in for the second column of a page with the +% publisher ID. If not needed in future releases, please provide this +% command and define it as \relax for backward compatibility +% v1.6b do not allow command to operate if the peer review option has been +% selected because \IEEEpubidadjcol will not be on the cover page. +% V1.7 do nothing if compsoc +\def\IEEEpubidadjcol{\ifCLASSOPTIONcompsoc\else\ifCLASSOPTIONpeerreview\else +\if@twocolumn\if@IEEEusingpubid\enlargethispage{-\@IEEEpubidpullup}\fi\fi\fi\fi} + +% Special thanks to Peter Wilson, Daniel Luecking, and the other +% gurus at comp.text.tex, for helping me to understand how best to +% implement the IEEEpubid command in LaTeX. + + + +%% Lockout some commands under various conditions + +% general purpose bit bucket +\newsavebox{\@IEEEtranrubishbin} + +% flags to prevent multiple warning messages +\newif\if@IEEEWARNthanks +\newif\if@IEEEWARNIEEEPARstart +\newif\if@IEEEWARNIEEEbiography +\newif\if@IEEEWARNIEEEbiographynophoto +\newif\if@IEEEWARNIEEEpubid +\newif\if@IEEEWARNIEEEpubidadjcol +\newif\if@IEEEWARNIEEEmembership +\newif\if@IEEEWARNIEEEaftertitletext +\@IEEEWARNthankstrue +\@IEEEWARNIEEEPARstarttrue +\@IEEEWARNIEEEbiographytrue +\@IEEEWARNIEEEbiographynophototrue +\@IEEEWARNIEEEpubidtrue +\@IEEEWARNIEEEpubidadjcoltrue +\@IEEEWARNIEEEmembershiptrue +\@IEEEWARNIEEEaftertitletexttrue + + +%% Lockout some commands when in various modes, but allow them to be restored if needed +%% +% save commands which might be locked out +% so that the user can later restore them if needed +\let\@IEEESAVECMDthanks\thanks +\let\@IEEESAVECMDIEEEPARstart\IEEEPARstart +\let\@IEEESAVECMDIEEEbiography\IEEEbiography +\let\@IEEESAVECMDendIEEEbiography\endIEEEbiography +\let\@IEEESAVECMDIEEEbiographynophoto\IEEEbiographynophoto +\let\@IEEESAVECMDendIEEEbiographynophoto\endIEEEbiographynophoto +\let\@IEEESAVECMDIEEEpubid\IEEEpubid +\let\@IEEESAVECMDIEEEpubidadjcol\IEEEpubidadjcol +\let\@IEEESAVECMDIEEEmembership\IEEEmembership +\let\@IEEESAVECMDIEEEaftertitletext\IEEEaftertitletext + + +% disable \IEEEPARstart when in draft mode +% This may have originally been done because the pre-V1.6 drop letter +% algorithm had problems with a non-unity baselinestretch +% At any rate, it seems too formal to have a drop letter in a draft +% paper. +\ifCLASSOPTIONdraftcls +\def\IEEEPARstart#1#2{#1#2\if@IEEEWARNIEEEPARstart\typeout{** ATTENTION: \noexpand\IEEEPARstart + is disabled in draft mode (line \the\inputlineno).}\fi\global\@IEEEWARNIEEEPARstartfalse} +\fi +% and for technotes +\ifCLASSOPTIONtechnote +\def\IEEEPARstart#1#2{#1#2\if@IEEEWARNIEEEPARstart\typeout{** WARNING: \noexpand\IEEEPARstart + is locked out for technotes (line \the\inputlineno).}\fi\global\@IEEEWARNIEEEPARstartfalse} +\fi + + +% lockout unneeded commands when in conference mode +\ifCLASSOPTIONconference +% when locked out, \thanks, \IEEEbiography, \IEEEbiographynophoto, \IEEEpubid, +% \IEEEmembership and \IEEEaftertitletext will all swallow their given text. +% \IEEEPARstart will output a normal character instead +% warn the user about these commands only once to prevent the console screen +% from filling up with redundant messages +\def\thanks#1{\if@IEEEWARNthanks\typeout{** WARNING: \noexpand\thanks + is locked out when in conference mode (line \the\inputlineno).}\fi\global\@IEEEWARNthanksfalse} +\def\IEEEPARstart#1#2{#1#2\if@IEEEWARNIEEEPARstart\typeout{** WARNING: \noexpand\IEEEPARstart + is locked out when in conference mode (line \the\inputlineno).}\fi\global\@IEEEWARNIEEEPARstartfalse} + + +% LaTeX treats environments and commands with optional arguments differently. +% the actual ("internal") command is stored as \\commandname +% (accessed via \csname\string\commandname\endcsname ) +% the "external" command \commandname is a macro with code to determine +% whether or not the optional argument is presented and to provide the +% default if it is absent. So, in order to save and restore such a command +% we would have to save and restore \\commandname as well. But, if LaTeX +% ever changes the way it names the internal names, the trick would break. +% Instead let us just define a new environment so that the internal +% name can be left undisturbed. +\newenvironment{@IEEEbogusbiography}[2][]{\if@IEEEWARNIEEEbiography\typeout{** WARNING: \noexpand\IEEEbiography + is locked out when in conference mode (line \the\inputlineno).}\fi\global\@IEEEWARNIEEEbiographyfalse% +\setbox\@IEEEtranrubishbin\vbox\bgroup}{\egroup\relax} +% and make biography point to our bogus biography +\let\IEEEbiography=\@IEEEbogusbiography +\let\endIEEEbiography=\end@IEEEbogusbiography + +\renewenvironment{IEEEbiographynophoto}[1]{\if@IEEEWARNIEEEbiographynophoto\typeout{** WARNING: \noexpand\IEEEbiographynophoto + is locked out when in conference mode (line \the\inputlineno).}\fi\global\@IEEEWARNIEEEbiographynophotofalse% +\setbox\@IEEEtranrubishbin\vbox\bgroup}{\egroup\relax} + +\def\IEEEpubid#1{\if@IEEEWARNIEEEpubid\typeout{** WARNING: \noexpand\IEEEpubid + is locked out when in conference mode (line \the\inputlineno).}\fi\global\@IEEEWARNIEEEpubidfalse} +\def\IEEEpubidadjcol{\if@IEEEWARNIEEEpubidadjcol\typeout{** WARNING: \noexpand\IEEEpubidadjcol + is locked out when in conference mode (line \the\inputlineno).}\fi\global\@IEEEWARNIEEEpubidadjcolfalse} +\def\IEEEmembership#1{\if@IEEEWARNIEEEmembership\typeout{** WARNING: \noexpand\IEEEmembership + is locked out when in conference mode (line \the\inputlineno).}\fi\global\@IEEEWARNIEEEmembershipfalse} +\def\IEEEaftertitletext#1{\if@IEEEWARNIEEEaftertitletext\typeout{** WARNING: \noexpand\IEEEaftertitletext + is locked out when in conference mode (line \the\inputlineno).}\fi\global\@IEEEWARNIEEEaftertitletextfalse} +\fi + + +% provide a way to restore the commands that are locked out +\def\IEEEoverridecommandlockouts{% +\typeout{** ATTENTION: Overriding command lockouts (line \the\inputlineno).}% +\let\thanks\@IEEESAVECMDthanks% +\let\IEEEPARstart\@IEEESAVECMDIEEEPARstart% +\let\IEEEbiography\@IEEESAVECMDIEEEbiography% +\let\endIEEEbiography\@IEEESAVECMDendIEEEbiography% +\let\IEEEbiographynophoto\@IEEESAVECMDIEEEbiographynophoto% +\let\endIEEEbiographynophoto\@IEEESAVECMDendIEEEbiographynophoto% +\let\IEEEpubid\@IEEESAVECMDIEEEpubid% +\let\IEEEpubidadjcol\@IEEESAVECMDIEEEpubidadjcol% +\let\IEEEmembership\@IEEESAVECMDIEEEmembership% +\let\IEEEaftertitletext\@IEEESAVECMDIEEEaftertitletext} + + + +% need a backslash character for typeout output +{\catcode`\|=0 \catcode`\\=12 +|xdef|@IEEEbackslash{\}} + + +% hook to allow easy disabling of all legacy warnings +\def\@IEEElegacywarn#1#2{\typeout{** ATTENTION: \@IEEEbackslash #1 is deprecated (line \the\inputlineno). +Use \@IEEEbackslash #2 instead.}} + + +% provide for legacy commands +\def\authorblockA{\@IEEElegacywarn{authorblockA}{IEEEauthorblockA}\IEEEauthorblockA} +\def\authorblockN{\@IEEElegacywarn{authorblockN}{IEEEauthorblockN}\IEEEauthorblockN} +\def\authorrefmark{\@IEEElegacywarn{authorrefmark}{IEEEauthorrefmark}\IEEEauthorrefmark} +\def\PARstart{\@IEEElegacywarn{PARstart}{IEEEPARstart}\IEEEPARstart} +\def\pubid{\@IEEElegacywarn{pubid}{IEEEpubid}\IEEEpubid} +\def\pubidadjcol{\@IEEElegacywarn{pubidadjcol}{IEEEpubidadjcol}\IEEEpubidadjcol} +\def\QED{\@IEEElegacywarn{QED}{IEEEQED}\IEEEQED} +\def\QEDclosed{\@IEEElegacywarn{QEDclosed}{IEEEQEDclosed}\IEEEQEDclosed} +\def\QEDopen{\@IEEElegacywarn{QEDopen}{IEEEQEDopen}\IEEEQEDopen} +\def\specialpapernotice{\@IEEElegacywarn{specialpapernotice}{IEEEspecialpapernotice}\IEEEspecialpapernotice} + + + +% provide for legacy environments +\def\biography{\@IEEElegacywarn{biography}{IEEEbiography}\IEEEbiography} +\def\biographynophoto{\@IEEElegacywarn{biographynophoto}{IEEEbiographynophoto}\IEEEbiographynophoto} +\def\keywords{\@IEEElegacywarn{keywords}{IEEEkeywords}\IEEEkeywords} +\def\endbiography{\endIEEEbiography} +\def\endbiographynophoto{\endIEEEbiographynophoto} +\def\endkeywords{\endIEEEkeywords} + + +% provide for legacy IED commands/lengths when possible +\let\labelindent\IEEElabelindent +\def\calcleftmargin{\@IEEElegacywarn{calcleftmargin}{IEEEcalcleftmargin}\IEEEcalcleftmargin} +\def\setlabelwidth{\@IEEElegacywarn{setlabelwidth}{IEEEsetlabelwidth}\IEEEsetlabelwidth} +\def\usemathlabelsep{\@IEEElegacywarn{usemathlabelsep}{IEEEusemathlabelsep}\IEEEusemathlabelsep} +\def\iedlabeljustifyc{\@IEEElegacywarn{iedlabeljustifyc}{IEEEiedlabeljustifyc}\IEEEiedlabeljustifyc} +\def\iedlabeljustifyl{\@IEEElegacywarn{iedlabeljustifyl}{IEEEiedlabeljustifyl}\IEEEiedlabeljustifyl} +\def\iedlabeljustifyr{\@IEEElegacywarn{iedlabeljustifyr}{IEEEiedlabeljustifyr}\IEEEiedlabeljustifyr} + + + +% let \proof use the IEEEtran version even after amsthm is loaded +% \proof is now deprecated in favor of \IEEEproof +\AtBeginDocument{\def\proof{\@IEEElegacywarn{proof}{IEEEproof}\IEEEproof}\def\endproof{\endIEEEproof}} + +% V1.7 \overrideIEEEmargins is no longer supported. +\def\overrideIEEEmargins{% +\typeout{** WARNING: \string\overrideIEEEmargins \space no longer supported (line \the\inputlineno).}% +\typeout{** Use the \string\CLASSINPUTinnersidemargin, \string\CLASSINPUToutersidemargin \space controls instead.}} + + +\endinput + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%% End of IEEEtran.cls %%%%%%%%%%%%%%%%%%%%%%%%%%%% +% That's all folks! + diff --git a/cometbft/v0.38/spec/consensus/consensus-paper/algorithmicplus.sty b/cometbft/v0.38/spec/consensus/consensus-paper/algorithmicplus.sty new file mode 100644 index 00000000..de7ca01e --- /dev/null +++ b/cometbft/v0.38/spec/consensus/consensus-paper/algorithmicplus.sty @@ -0,0 +1,195 @@ +% ALGORITHMICPLUS STYLE +% for LaTeX version 2e +% Original ``algorithmic.sty'' by -- 1994 Peter Williams +% Bug fix (13 July 2004) by Arnaud Giersch +% Includes ideas from 'algorithmicext' by Martin Biely +% and 'distribalgo' by Xavier Defago +% Modifications: Martin Hutle +% +% This style file is free software; you can redistribute it and/or +% modify it under the terms of the GNU Lesser General Public +% License as published by the Free Software Foundation; either +% version 2 of the License, or (at your option) any later version. +% +% This style file is distributed in the hope that it will be useful, +% but WITHOUT ANY WARRANTY; without even the implied warranty of +% MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +% Lesser General Public License for more details. +% +% You should have received a copy of the GNU Lesser General Public +% License along with this style file; if not, write to the +% Free Software Foundation, Inc., 59 Temple Place - Suite 330, +% Boston, MA 02111-1307, USA. +% +\NeedsTeXFormat{LaTeX2e} +\ProvidesPackage{algorithmicplus} +\typeout{Document Style `algorithmicplus' - environment, replaces `algorithmic'} +% +\RequirePackage{ifthen} +\RequirePackage{calc} +\newboolean{ALC@noend} +\setboolean{ALC@noend}{false} +\newcounter{ALC@line} +\newcounter{ALC@rem} +\newcounter{ALC@depth} +\newcounter{ALCPLUS@lastline} +\newlength{\ALC@tlm} +% +\DeclareOption{noend}{\setboolean{ALC@noend}{true}} +% +\ProcessOptions +% +% ALGORITHMIC +\newcommand{\algorithmiclnosize}{\small} +\newcommand{\algorithmiclnofont}{\tt} +\newcommand{\algorithmiclnodelimiter}{:} +% +\newcommand{\algorithmicrequire}{\textbf{Require:}} +\newcommand{\algorithmicensure}{\textbf{Ensure:}} +\newcommand{\algorithmiccomment}[1]{\{#1\}} +\newcommand{\algorithmicend}{\textbf{end}} +\newcommand{\algorithmicif}{\textbf{if}} +\newcommand{\algorithmicthen}{\textbf{then}} +\newcommand{\algorithmicelse}{\textbf{else}} +\newcommand{\algorithmicelsif}{\algorithmicelse\ \algorithmicif} +\newcommand{\algorithmicendif}{\algorithmicend\ \algorithmicif} +\newcommand{\algorithmicfor}{\textbf{for}} +\newcommand{\algorithmicforall}{\textbf{for all}} +\newcommand{\algorithmicdo}{\textbf{do}} +\newcommand{\algorithmicendfor}{\algorithmicend\ \algorithmicfor} +\newcommand{\algorithmicwhile}{\textbf{while}} +\newcommand{\algorithmicendwhile}{\algorithmicend\ \algorithmicwhile} +\newcommand{\algorithmicloop}{\textbf{loop}} +\newcommand{\algorithmicendloop}{\algorithmicend\ \algorithmicloop} +\newcommand{\algorithmicrepeat}{\textbf{repeat}} +\newcommand{\algorithmicuntil}{\textbf{until}} +\def\ALC@item[#1]{% +\if@noparitem \@donoparitem + \else \if@inlabel \indent \par \fi + \ifhmode \unskip\unskip \par \fi + \if@newlist \if@nobreak \@nbitem \else + \addpenalty\@beginparpenalty + \addvspace\@topsep \addvspace{-\parskip}\fi + \else \addpenalty\@itempenalty \addvspace\itemsep + \fi + \global\@inlabeltrue +\fi +\everypar{\global\@minipagefalse\global\@newlistfalse + \if@inlabel\global\@inlabelfalse \hskip -\parindent \box\@labels + \penalty\z@ \fi + \everypar{}}\global\@nobreakfalse +\if@noitemarg \@noitemargfalse \if@nmbrlist \refstepcounter{\@listctr}\fi \fi +\sbox\@tempboxa{\makelabel{#1}}% +\global\setbox\@labels + \hbox{\unhbox\@labels \hskip \itemindent + \hskip -\labelwidth \hskip -\ALC@tlm + \ifdim \wd\@tempboxa >\labelwidth + \box\@tempboxa + \else \hbox to\labelwidth {\unhbox\@tempboxa}\fi + \hskip \ALC@tlm}\ignorespaces} +% +\newenvironment{algorithmic}[1][0]{ +\setcounter{ALC@depth}{\@listdepth}% +\let\@listdepth\c@ALC@depth% +\let\@item\ALC@item + \newcommand{\ALC@lno}{% +\ifthenelse{\equal{\arabic{ALC@rem}}{0}} +{{\algorithmiclnosize\algorithmiclnofont \arabic{ALC@line}\algorithmiclnodelimiter}}{}% +} +\let\@listii\@listi +\let\@listiii\@listi +\let\@listiv\@listi +\let\@listv\@listi +\let\@listvi\@listi +\let\@listvii\@listi + \newenvironment{ALC@g}{ + \begin{list}{\ALC@lno}{ \itemsep\z@ \itemindent\z@ + \listparindent\z@ \rightmargin\z@ + \topsep\z@ \partopsep\z@ \parskip\z@\parsep\z@ + \leftmargin 1em + \addtolength{\ALC@tlm}{\leftmargin} + } + } + {\end{list}} + \newcommand{\ALC@it}{\refstepcounter{ALC@line}\addtocounter{ALC@rem}{1}\ifthenelse{\equal{\arabic{ALC@rem}}{#1}}{\setcounter{ALC@rem}{0}}{}\item} + \newcommand{\ALC@com}[1]{\ifthenelse{\equal{##1}{default}}% +{}{\ \algorithmiccomment{##1}}} + \newcommand{\REQUIRE}{\item[\algorithmicrequire]} + \newcommand{\ENSURE}{\item[\algorithmicensure]} + \newcommand{\STATE}{\ALC@it} + \newcommand{\COMMENT}[1]{\algorithmiccomment{##1}} + \newenvironment{ALC@if}{\begin{ALC@g}}{\end{ALC@g}} + \newenvironment{ALC@for}{\begin{ALC@g}}{\end{ALC@g}} + \newenvironment{ALC@whl}{\begin{ALC@g}}{\end{ALC@g}} + \newenvironment{ALC@loop}{\begin{ALC@g}}{\end{ALC@g}} + \newenvironment{ALC@rpt}{\begin{ALC@g}}{\end{ALC@g}} + \renewcommand{\\}{\@centercr} + \newcommand{\IF}[2][default]{\ALC@it\algorithmicif\ ##2\ \algorithmicthen% +\ALC@com{##1}\begin{ALC@if}} + \newcommand{\ELSE}[1][default]{\end{ALC@if}\ALC@it\algorithmicelse% +\ALC@com{##1}\begin{ALC@if}} + \newcommand{\ELSIF}[2][default]% +{\end{ALC@if}\ALC@it\algorithmicelsif\ ##2\ \algorithmicthen% +\ALC@com{##1}\begin{ALC@if}} + \newcommand{\FOR}[2][default]{\ALC@it\algorithmicfor\ ##2\ \algorithmicdo% +\ALC@com{##1}\begin{ALC@for}} + \newcommand{\FORALL}[2][default]{\ALC@it\algorithmicforall\ ##2\ % +\algorithmicdo% +\ALC@com{##1}\begin{ALC@for}} + \newcommand{\WHILE}[2][default]{\ALC@it\algorithmicwhile\ ##2\ % +\algorithmicdo% +\ALC@com{##1}\begin{ALC@whl}} + \newcommand{\LOOP}[1][default]{\ALC@it\algorithmicloop% +\ALC@com{##1}\begin{ALC@loop}} + \newcommand{\REPEAT}[1][default]{\ALC@it\algorithmicrepeat% +\ALC@com{##1}\begin{ALC@rpt}} + \newcommand{\UNTIL}[1]{\end{ALC@rpt}\ALC@it\algorithmicuntil\ ##1} + \ifthenelse{\boolean{ALC@noend}}{ + \newcommand{\ENDIF}{\end{ALC@if}} + \newcommand{\ENDFOR}{\end{ALC@for}} + \newcommand{\ENDWHILE}{\end{ALC@whl}} + \newcommand{\ENDLOOP}{\end{ALC@loop}} + }{ + \newcommand{\ENDIF}{\end{ALC@if}\ALC@it\algorithmicendif} + \newcommand{\ENDFOR}{\end{ALC@for}\ALC@it\algorithmicendfor} + \newcommand{\ENDWHILE}{\end{ALC@whl}\ALC@it\algorithmicendwhile} + \newcommand{\ENDLOOP}{\end{ALC@loop}\ALC@it\algorithmicendloop} + } + \renewcommand{\@toodeep}{} + \begin{list}{\ALC@lno}{\setcounter{ALC@line}{0}\setcounter{ALC@rem}{0}% + \itemsep\z@ \itemindent\z@ \listparindent\z@% + \partopsep\z@ \parskip\z@ \parsep\z@% + \labelsep 0.5em \topsep 0.2em% +\ifthenelse{\equal{#1}{0}} + {\labelwidth 0.5em } + {\labelwidth 1.2em } +\leftmargin\labelwidth \addtolength{\leftmargin}{\labelsep} + \ALC@tlm\labelsep + } +} +{% +\setcounter{ALCPLUS@lastline}{\value{ALC@line}}% +\end{list}} + +\newcommand{\continuecounting}{\setcounter{ALC@line}{\value{ALCPLUS@lastline}}} +\newcommand{\startcounting}[1]{\setcounter{ALC@line}{#1}\addtocounter{ALC@line}{-1}} + +\newcommand{\EMPTY}{\item[]} +\newcommand{\SPACE}{\vspace{3mm}} +\newcommand{\SHORTSPACE}{\vspace{1mm}} +\newcommand{\newlinetag}[3]{\newcommand{#1}[#2]{\item[#3]}} +\newcommand{\newconstruct}[5]{% + \newenvironment{ALC@\string#1}{\begin{ALC@g}}{\end{ALC@g}} + \newcommand{#1}[2][default]{\ALC@it#2\ ##2\ #3% + \ALC@com{##1}\begin{ALC@\string#1}} + \ifthenelse{\boolean{ALC@noend}}{ + \newcommand{#4}{\end{ALC@\string#1}} + }{ + \newcommand{#4}{\end{ALC@\string#1}\ALC@it#5} + } +} + +\newconstruct{\INDENT}{}{}{\ENDINDENT}{} + +\newcommand{\setlinenosize}[1]{\renewcommand{\algorithmiclnosize}{#1}} +\newcommand{\setlinenofont}[1]{\renewcommand{\algorithmiclnofont}{#1}} diff --git a/cometbft/v0.38/spec/consensus/consensus-paper/conclusion.tex b/cometbft/v0.38/spec/consensus/consensus-paper/conclusion.tex new file mode 100644 index 00000000..dd17ccf4 --- /dev/null +++ b/cometbft/v0.38/spec/consensus/consensus-paper/conclusion.tex @@ -0,0 +1,16 @@ +\section{Conclusion} \label{sec:conclusion} + +We have proposed a new Byzantine-fault tolerant consensus algorithm that is the +core of the Tendermint BFT SMR platform. The algorithm is designed for the wide +area network with high number of mutually distrusted nodes that communicate +over gossip based peer-to-peer network. It has only a single mode of execution +and the communication pattern is very similar to the "normal" case of the +state-of-the art PBFT algorithm. The algorithm ensures termination with a novel +mechanism that takes advantage of the gossip based communication between nodes. +The proposed algorithm and the proofs are simple and elegant, and we believe +that this makes it easier to understand and implement correctly. + +\section*{Acknowledgment} + +We would like to thank Anton Kaliaev, Ismail Khoffi and Dahlia Malkhi for comments on an earlier version of the paper. We also want to thank Marko Vukolic, Ming Chuan Lin, Maria Potop-Butucaru, Sara Tucci, Antonella Del Pozzo and Yackolley Amoussou-Guenou for pointing out the liveness issues +in the previous version of the algorithm. Finally, we want to thank the Tendermint team members and all project contributors for making Tendermint such a great platform. diff --git a/cometbft/v0.38/spec/consensus/consensus-paper/consensus.tex b/cometbft/v0.38/spec/consensus/consensus-paper/consensus.tex new file mode 100644 index 00000000..3265b61c --- /dev/null +++ b/cometbft/v0.38/spec/consensus/consensus-paper/consensus.tex @@ -0,0 +1,397 @@ + +\section{Tendermint consensus algorithm} \label{sec:tendermint} + +\newcommand\Disseminate{\textbf{Disseminate}} + +\newcommand\Proposal{\mathsf{PROPOSAL}} +\newcommand\ProposalPart{\mathsf{PROPOSAL\mbox{-}PART}} +\newcommand\PrePrepare{\mathsf{INIT}} \newcommand\Prevote{\mathsf{PREVOTE}} +\newcommand\Precommit{\mathsf{PRECOMMIT}} +\newcommand\Decision{\mathsf{DECISION}} + +\newcommand\ViewChange{\mathsf{VC}} +\newcommand\ViewChangeAck{\mathsf{VC\mbox{-}ACK}} +\newcommand\NewPrePrepare{\mathsf{VC\mbox{-}INIT}} +\newcommand\coord{\mathsf{proposer}} + +\newcommand\newHeight{newHeight} \newcommand\newRound{newRound} +\newcommand\nil{nil} \newcommand\id{id} \newcommand{\propose}{propose} +\newcommand\prevote{prevote} \newcommand\prevoteWait{prevoteWait} +\newcommand\precommit{precommit} \newcommand\precommitWait{precommitWait} +\newcommand\commit{commit} + +\newcommand\timeoutPropose{timeoutPropose} +\newcommand\timeoutPrevote{timeoutPrevote} +\newcommand\timeoutPrecommit{timeoutPrecommit} +\newcommand\proofOfLocking{proof\mbox{-}of\mbox{-}locking} + +\begin{algorithm}[htb!] \def\baselinestretch{1} \scriptsize\raggedright + \begin{algorithmic}[1] + \SHORTSPACE + \INIT{} + \STATE $h_p := 0$ + \COMMENT{current height, or consensus instance we are currently executing} + \STATE $round_p := 0$ \COMMENT{current round number} + \STATE $step_p \in \set{\propose, \prevote, \precommit}$ + \STATE $decision_p[] := nil$ + \STATE $lockedValue_p := nil$ + \STATE $lockedRound_p := -1$ + \STATE $validValue_p := nil$ + \STATE $validRound_p := -1$ + \ENDINIT + \SHORTSPACE + \STATE \textbf{upon} start \textbf{do} $StartRound(0)$ + \SHORTSPACE + \FUNCTION{$StartRound(round)$} \label{line:tab:startRound} + \STATE $round_p \assign round$ + \STATE $step_p \assign \propose$ + \IF{$\coord(h_p, round_p) = p$} + \IF{$validValue_p \neq \nil$} \label{line:tab:isThereLockedValue} + \STATE $proposal \assign validValue_p$ \ELSE \STATE $proposal \assign + getValue()$ + \label{line:tab:getValidValue} + \ENDIF + \STATE \Broadcast\ $\li{\Proposal,h_p, round_p, proposal, validRound_p}$ + \label{line:tab:send-proposal} + \ELSE + \STATE \textbf{schedule} $OnTimeoutPropose(h_p, + round_p)$ to be executed \textbf{after} $\timeoutPropose(round_p)$ + \ENDIF + \ENDFUNCTION + + \SPACE + \UPON{$\li{\Proposal,h_p,round_p, v, -1}$ \From\ $\coord(h_p,round_p)$ + \With\ $step_p = \propose$} \label{line:tab:recvProposal} + \IF{$valid(v) \wedge (lockedRound_p = -1 \vee lockedValue_p = v$)} + \label{line:tab:accept-proposal-2} + \STATE \Broadcast \ $\li{\Prevote,h_p,round_p,id(v)}$ + \label{line:tab:prevote-proposal} + \ELSE + \label{line:tab:acceptProposal1} + \STATE \Broadcast \ $\li{\Prevote,h_p,round_p,\nil}$ + \label{line:tab:prevote-nil} + \ENDIF + \STATE $step_p \assign \prevote$ \label{line:tab:setStateToPrevote1} + \ENDUPON + + \SPACE + \UPON{$\li{\Proposal,h_p,round_p, v, vr}$ \From\ $\coord(h_p,round_p)$ + \textbf{AND} $2f+1$ $\li{\Prevote,h_p, vr,id(v)}$ \With\ $step_p = \propose \wedge (vr \ge 0 \wedge vr < round_p)$} + \label{line:tab:acceptProposal} + \IF{$valid(v) \wedge (lockedRound_p \le vr + \vee lockedValue_p = v)$} \label{line:tab:cond-prevote-higher-proposal} + \STATE \Broadcast \ $\li{\Prevote,h_p,round_p,id(v)}$ + \label{line:tab:prevote-higher-proposal} + \ELSE + \label{line:tab:acceptProposal2} + \STATE \Broadcast \ $\li{\Prevote,h_p,round_p,\nil}$ + \label{line:tab:prevote-nil2} + \ENDIF + \STATE $step_p \assign \prevote$ \label{line:tab:setStateToPrevote3} + \ENDUPON + + \SPACE + \UPON{$2f+1$ $\li{\Prevote,h_p, round_p,*}$ \With\ $step_p = \prevote$ for the first time} + \label{line:tab:recvAny2/3Prevote} + \STATE \textbf{schedule} $OnTimeoutPrevote(h_p, round_p)$ to be executed \textbf{after} $\timeoutPrevote(round_p)$ \label{line:tab:timeoutPrevote} + \ENDUPON + + \SPACE + \UPON{$\li{\Proposal,h_p,round_p, v, *}$ \From\ $\coord(h_p,round_p)$ + \textbf{AND} $2f+1$ $\li{\Prevote,h_p, round_p,id(v)}$ \With\ $valid(v) \wedge step_p \ge \prevote$ for the first time} + \label{line:tab:recvPrevote} + \IF{$step_p = \prevote$} + \STATE $lockedValue_p \assign v$ \label{line:tab:setLockedValue} + \STATE $lockedRound_p \assign round_p$ \label{line:tab:setLockedRound} + \STATE \Broadcast \ $\li{\Precommit,h_p,round_p,id(v))}$ + \label{line:tab:precommit-v} + \STATE $step_p \assign \precommit$ \label{line:tab:setStateToCommit} + \ENDIF + \STATE $validValue_p \assign v$ \label{line:tab:setValidRound} + \STATE $validRound_p \assign round_p$ \label{line:tab:setValidValue} + \ENDUPON + + \SHORTSPACE + \UPON{$2f+1$ $\li{\Prevote,h_p,round_p, \nil}$ + \With\ $step_p = \prevote$} + \STATE \Broadcast \ $\li{\Precommit,h_p,round_p, \nil}$ + \label{line:tab:precommit-v-1} + \STATE $step_p \assign \precommit$ + \ENDUPON + + \SPACE + \UPON{$2f+1$ $\li{\Precommit,h_p,round_p,*}$ for the first time} + \label{line:tab:startTimeoutPrecommit} + \STATE \textbf{schedule} $OnTimeoutPrecommit(h_p, round_p)$ to be executed \textbf{after} $\timeoutPrecommit(round_p)$ + + \ENDUPON + + \SPACE + \UPON{$\li{\Proposal,h_p,r, v, *}$ \From\ $\coord(h_p,r)$ \textbf{AND} + $2f+1$ $\li{\Precommit,h_p,r,id(v)}$ \With\ $decision_p[h_p] = \nil$} + \label{line:tab:onDecideRule} + \IF{$valid(v)$} \label{line:tab:validDecisionValue} + \STATE $decision_p[h_p] = v$ \label{line:tab:decide} + \STATE$h_p \assign h_p + 1$ \label{line:tab:increaseHeight} + \STATE reset $lockedRound_p$, $lockedValue_p$, $validRound_p$ and $validValue_p$ to initial values + and empty message log + \STATE $StartRound(0)$ + \ENDIF + \ENDUPON + + \SHORTSPACE + \UPON{$f+1$ $\li{*,h_p,round, *, *}$ \textbf{with} $round > round_p$} + \label{line:tab:skipRounds} + \STATE $StartRound(round)$ \label{line:tab:nextRound2} + \ENDUPON + + \SHORTSPACE + \FUNCTION{$OnTimeoutPropose(height,round)$} \label{line:tab:onTimeoutPropose} + \IF{$height = h_p \wedge round = round_p \wedge step_p = \propose$} + \STATE \Broadcast \ $\li{\Prevote,h_p,round_p, \nil}$ + \label{line:tab:prevote-nil-on-timeout} + \STATE $step_p \assign \prevote$ + \ENDIF + \ENDFUNCTION + + \SHORTSPACE + \FUNCTION{$OnTimeoutPrevote(height,round)$} \label{line:tab:onTimeoutPrevote} + \IF{$height = h_p \wedge round = round_p \wedge step_p = \prevote$} + \STATE \Broadcast \ $\li{\Precommit,h_p,round_p,\nil}$ + \label{line:tab:precommit-nil-onTimeout} + \STATE $step_p \assign \precommit$ + \ENDIF + \ENDFUNCTION + + \SHORTSPACE + \FUNCTION{$OnTimeoutPrecommit(height,round)$} \label{line:tab:onTimeoutPrecommit} + \IF{$height = h_p \wedge round = round_p$} + \STATE $StartRound(round_p + 1)$ \label{line:tab:nextRound} + \ENDIF + \ENDFUNCTION + \end{algorithmic} \caption{Tendermint consensus algorithm} + \label{alg:tendermint} +\end{algorithm} + +In this section we present the Tendermint Byzantine fault-tolerant consensus +algorithm. The algorithm is specified by the pseudo-code shown in +Algorithm~\ref{alg:tendermint}. We present the algorithm as a set of \emph{upon +rules} that are executed atomically\footnote{In case several rules are active +at the same time, the first rule to be executed is picked randomly. The +correctness of the algorithm does not depend on the order in which rules are +executed.}. We assume that processes exchange protocol messages using a gossip +protocol and that both sent and received messages are stored in a local message +log for every process. An upon rule is triggered once the message log contains +messages such that the corresponding condition evaluates to $\tt{true}$. The +condition that assumes reception of $X$ messages of a particular type and +content denotes reception of messages whose senders have aggregate voting power at +least equal to $X$. For example, the condition $2f+1$ $\li{\Precommit,h_p,r,id(v)}$, +evaluates to true upon reception of $\Precommit$ messages for height $h_p$, +a round $r$ and with value equal to $id(v)$ whose senders have aggregate voting +power at least equal to $2f+1$. Some of the rules ends with "for the first time" constraint +to denote that it is triggered only the first time a corresponding condition evaluates +to $\tt{true}$. This is because those rules do not always change the state of algorithm +variables so without this constraint, the algorithm could keep +executing those rules forever. The variables with index $p$ are process local state +variables, while variables without index $p$ are value placeholders. The sign +$*$ denotes any value. + +We denote with $n$ the total voting power of processes in the system, and we +assume that the total voting power of faulty processes in the system is bounded +with a system parameter $f$. The algorithm assumes that $n > 3f$, i.e., it +requires that the total voting power of faulty processes is smaller than one +third of the total voting power. For simplicity we present the algorithm for +the case $n = 3f + 1$. + +The algorithm proceeds in rounds, where each round has a dedicated +\emph{proposer}. The mapping of rounds to proposers is known to all processes +and is given as a function $\coord(h, round)$, returning the proposer for +the round $round$ in the consensus instance $h$. We +assume that the proposer selection function is weighted round-robin, where +processes are rotated proportional to their voting power\footnote{A validator +with more voting power is selected more frequently, proportional to its power. +More precisely, during a sequence of rounds of size $n$, every process is +proposer in a number of rounds equal to its voting power.}. +The internal protocol state transitions are triggered by message reception and +by expiration of timeouts. There are three timeouts in Algorithm \ref{alg:tendermint}: +$\timeoutPropose$, $\timeoutPrevote$ and $\timeoutPrecommit$. +The timeouts prevent the algorithm from blocking and +waiting forever for some condition to be true, ensure that processes continuously +transition between rounds, and guarantee that eventually (after GST) communication +between correct processes is timely and reliable so they can decide. +The last role is achieved by increasing the timeouts with every new round $r$, +i.e, $timeoutX(r) = initTimeoutX + r*timeoutDelta$; +they are reset for every new height (consensus +instance). + +Processes exchange the following messages in Tendermint: $\Proposal$, +$\Prevote$ and $\Precommit$. The $\Proposal$ message is used by the proposer of +the current round to suggest a potential decision value, while $\Prevote$ and +$\Precommit$ are votes for a proposed value. According to the classification of +consensus algorithms from \cite{RMS10:dsn}, Tendermint, like PBFT +\cite{CL02:tcs} and DLS \cite{DLS88:jacm}, belongs to class 3, so it requires +two voting steps (three communication exchanges in total) to decide a value. +The Tendermint consensus algorithm is designed for the blockchain context where +the value to decide is a block of transactions (ie. it is potentially quite +large, consisting of many transactions). Therefore, in the Algorithm +\ref{alg:tendermint} (similar as in \cite{CL02:tcs}) we are explicit about +sending a value (block of transactions) and a small, constant size value id (a +unique value identifier, normally a hash of the value, i.e., if $\id(v) = +\id(v')$, then $v=v'$). The $\Proposal$ message is the only one carrying the +value; $\Prevote$ and $\Precommit$ messages carry the value id. A correct +process decides on a value $v$ in Tendermint upon receiving the $\Proposal$ for +$v$ and $2f+1$ voting-power equivalent $\Precommit$ messages for $\id(v)$ in +some round $r$. In order to send $\Precommit$ message for $v$ in a round $r$, a +correct process waits to receive the $\Proposal$ and $2f+1$ of the +corresponding $\Prevote$ messages in the round $r$. Otherwise, +it sends $\Precommit$ message with a special $\nil$ value. +This ensures that correct processes can $\Precommit$ only a +single value (or $\nil$) in a round. As +proposers may be faulty, the proposed value is treated by correct processes as +a suggestion (it is not blindly accepted), and a correct process tells others +if it accepted the $\Proposal$ for value $v$ by sending $\Prevote$ message for +$\id(v)$; otherwise it sends $\Prevote$ message with the special $\nil$ value. + +Every process maintains the following variables in the Algorithm +\ref{alg:tendermint}: $step$, $lockedValue$, $lockedRound$, $validValue$ and +$validRound$. The $step$ denotes the current state of the internal Tendermint +state machine, i.e., it reflects the stage of the algorithm execution in the +current round. The $lockedValue$ stores the most recent value (with respect to +a round number) for which a $\Precommit$ message has been sent. The +$lockedRound$ is the last round in which the process sent a $\Precommit$ +message that is not $\nil$. We also say that a correct process locks a value +$v$ in a round $r$ by setting $lockedValue = v$ and $lockedRound = r$ before +sending $\Precommit$ message for $\id(v)$. As a correct process can decide a +value $v$ only if $2f+1$ $\Precommit$ messages for $\id(v)$ are received, this +implies that a possible decision value is a value that is locked by at least +$f+1$ voting power equivalent of correct processes. Therefore, any value $v$ +for which $\Proposal$ and $2f+1$ of the corresponding $\Prevote$ messages are +received in some round $r$ is a \emph{possible decision} value. The role of the +$validValue$ variable is to store the most recent possible decision value; the +$validRound$ is the last round in which $validValue$ is updated. Apart from +those variables, a process also stores the current consensus instance ($h_p$, +called \emph{height} in Tendermint), and the current round number ($round_p$) +and attaches them to every message. Finally, a process also stores an array of +decisions, $decision_p$ (Tendermint assumes a sequence of consensus instances, +one for each height). + +Every round starts by a proposer suggesting a value with the $\Proposal$ +message (see line \ref{line:tab:send-proposal}). In the initial round of each +height, the proposer is free to chose the value to suggest. In the +Algorithm~\ref{alg:tendermint}, a correct process obtains a value to propose +using an external function $getValue()$ that returns a valid value to +propose. In the following rounds, a correct proposer will suggest a new value +only if $validValue = \nil$; otherwise $validValue$ is proposed (see +lines~\ref{line:tab:isThereLockedValue}-\ref{line:tab:getValidValue}). +In addition to the value proposed, the $\Proposal$ message also +contains the $validRound$ so other processes are informed about the last round +in which the proposer observed $validValue$ as a possible decision value. +Note that if a correct proposer $p$ sends $validValue$ with the $validRound$ in the +$\Proposal$, this implies that the process $p$ received $\Proposal$ and the +corresponding $2f+1$ $\Prevote$ messages for $validValue$ in the round +$validRound$. +If a correct process sends $\Proposal$ message with $validValue$ ($validRound > -1$) +at time $t > GST$, by the \emph{Gossip communication} property, the +corresponding $\Proposal$ and the $\Prevote$ messages will be received by all +correct processes before time $t+\Delta$. Therefore, all correct processes will +be able to verify the correctness of the suggested value as it is supported by +the $\Proposal$ and the corresponding $2f+1$ voting power equivalent $\Prevote$ +messages. + +A correct process $p$ accepts the proposal for a value $v$ (send $\Prevote$ +for $id(v)$) if an external \emph{valid} function returns $true$ for the value +$v$, and if $p$ hasn't locked any value ($lockedRound = -1$) or $p$ has locked +the value $v$ ($lockedValue = v$); see the line +\ref{line:tab:accept-proposal-2}. In case the proposed pair is $(v,vr \ge 0)$ and a +correct process $p$ has locked some value, it will accept +$v$ if it is a more recent possible decision value\footnote{As +explained above, the possible decision value in a round $r$ is the one for +which $\Proposal$ and the corresponding $2f+1$ $\Prevote$ messages are received +for the round $r$.}, $vr > lockedRound_p$, or if $lockedValue = v$ +(see line~\ref{line:tab:cond-prevote-higher-proposal}). Otherwise, a correct +process will reject the proposal by sending $\Prevote$ message with $\nil$ +value. A correct process will send $\Prevote$ message with $\nil$ value also in +case $\timeoutPropose$ expired (it is triggered when a correct process starts a +new round) and a process has not sent $\Prevote$ message in the current round +yet (see the line \ref{line:tab:onTimeoutPropose}). + +If a correct process receives $\Proposal$ message for some value $v$ and $2f+1$ +$\Prevote$ messages for $\id(v)$, then it sends $\Precommit$ message with +$\id(v)$. Otherwise, it sends $\Precommit$ $\nil$. A correct process will send +$\Precommit$ message with $\nil$ value also in case $\timeoutPrevote$ expired +(it is started when a correct process sent $\Prevote$ message and received any +$2f+1$ $\Prevote$ messages) and a process has not sent $\Precommit$ message in +the current round yet (see the line \ref{line:tab:onTimeoutPrecommit}). A +correct process decides on some value $v$ if it receives in some round $r$ +$\Proposal$ message for $v$ and $2f+1$ $\Precommit$ messages with $\id(v)$ (see +the line \ref{line:tab:decide}). To prevent the algorithm from blocking and +waiting forever for this condition to be true, the Algorithm +\ref{alg:tendermint} relies on $\timeoutPrecommit$. It is triggered after a +process receives any set of $2f+1$ $\Precommit$ messages for the current round. +If the $\timeoutPrecommit$ expires and a process has not decided yet, the +process starts the next round (see the line \ref{line:tab:onTimeoutPrecommit}). +When a correct process $p$ decides, it starts the next consensus instance +(for the next height). The \emph{Gossip communication} property ensures +that $\Proposal$ and $2f+1$ $\Prevote$ messages that led $p$ to decide +are eventually received by all correct processes, so they will also decide. + +\subsection{Termination mechanism} + +Tendermint ensures termination by a novel mechanism that benefits from the +gossip based nature of communication (see \emph{Gossip communication} +property). It requires managing two additional variables, $validValue$ and +$validRound$ that are then used by the proposer during the propose step as +explained above. The $validValue$ and $validRound$ are updated to $v$ and $r$ +by a correct process in a round $r$ when the process receives valid $\Proposal$ +message for the value $v$ and the corresponding $2f+1$ $\Prevote$ messages for +$id(v)$ in the round $r$ (see the rule at line~\ref{line:tab:recvPrevote}). + +We now give briefly the intuition how managing and proposing $validValue$ +and $validRound$ ensures termination. Formal treatment is left for +Section~\ref{sec:proof}. + +The first thing to note is that during good period, because of the +\emph{Gossip communication} property, if a correct process $p$ locks a value +$v$ in some round $r$, all correct processes will update $validValue$ to $v$ +and $validRound$ to $r$ before the end of the round $r$ (we prove this formally +in the Section~\ref{sec:proof}). The intuition is that messages that led to $p$ +locking a value $v$ in the round $r$ will be gossiped to all correct processes +before the end of the round $r$, so it will update $validValue$ and +$validRound$ (the line~\ref{line:tab:recvPrevote}). Therefore, if a correct +process locks some value during good period, $validValue$ and $validRound$ are +updated by all correct processes so that the value proposed in the following +rounds will be acceptable by all correct processes. Note +that it could happen that during good period, no correct process locks a value, +but some correct process $q$ updates $validValue$ and $validRound$ during some +round. As no correct process locks a value in this case, $validValue_q$ and +$validRound_q$ will also be acceptable by all correct processes as +$validRound_q > lockedRound_c$ for every correct process $c$ and as the +\emph{Gossip communication} property ensures that the corresponding $\Prevote$ +messages that $q$ received in the round $validRound_q$ are received by all +correct processes $\Delta$ time later. + +Finally, it could happen that after GST, there is a long sequence of rounds in which +no correct process neither locks a value nor update $validValue$ and $validRound$. +In this case, during this sequence of rounds, the proposed value suggested by correct +processes was not accepted by all correct processes. Note that this sequence of rounds +is always finite as at the beginning of every +round there is at least a single correct process $c$ such that $validValue_c$ +and $validRound_c$ are acceptable by every correct process. This is true as +there exists a correct process $c$ such that for every other correct process +$p$, $validRound_c > lockedRound_p$ or $validValue_c = lockedValue_p$. This is +true as $c$ is the process that has locked a value in the most recent round +among all correct processes (or no correct process locked any value). Therefore, +eventually $c$ will be the proper in some round and the proposed value will be accepted +by all correct processes, terminating therefore this sequence of +rounds. + +Therefore, updating $validValue$ and $validRound$ variables, and the +\emph{Gossip communication} property, together ensures that eventually, during +the good period, there exists a round with a correct proposer whose proposed +value will be accepted by all correct processes, and all correct processes will +terminate in that round. Note that this mechanism, contrary to the common +termination mechanism illustrated in the +Figure~\ref{ch3:fig:coordinator-change}, does not require exchanging any +additional information in addition to messages already sent as part of what is +normally being called "normal" case. + diff --git a/cometbft/v0.38/spec/consensus/consensus-paper/definitions.tex b/cometbft/v0.38/spec/consensus/consensus-paper/definitions.tex new file mode 100644 index 00000000..454dd445 --- /dev/null +++ b/cometbft/v0.38/spec/consensus/consensus-paper/definitions.tex @@ -0,0 +1,126 @@ +\section{Definitions} \label{sec:definitions} + +\subsection{Model} + +We consider a system of processes that communicate by exchanging messages. +Processes can be correct or faulty, where a faulty process can behave in an +arbitrary way, i.e., we consider Byzantine faults. We assume that each process +has some amount of voting power (voting power of a process can be $0$). +Processes in our model are not part of a single administrative domain; +therefore we cannot enforce a direct network connectivity between all +processes. Instead, we assume that each process is connected to a subset of +processes called peers, such that there is an indirect communication channel +between all correct processes. Communication between processes is established +using a gossip protocol \cite{Dem1987:gossip}. + +Formally, we model the network communication using a variant of the \emph{partially +synchronous system model}~\cite{DLS88:jacm}: in all executions of the system +there is a bound $\Delta$ and an instant GST (Global Stabilization Time) such +that all communication among correct processes after GST is reliable and +$\Delta$-timely, i.e., if a correct process $p$ sends message $m$ at time $t +\ge GST$ to a correct process $q$, then $q$ will receive $m$ before $t + +\Delta$\footnote{Note that as we do not assume direct communication channels + among all correct processes, this implies that before the message $m$ + reaches $q$, it might pass through a number of correct processes that will +forward the message $m$ using gossip protocol towards $q$.}. +In addition to the standard \emph{partially + synchronous system model}~\cite{DLS88:jacm}, we assume an auxiliary property +that captures gossip-based nature of communication\footnote{The details of the Tendermint gossip protocol will be discussed in a separate + technical report. }: + + +\begin{itemize} \item \emph{Gossip communication:} If a correct process $p$ + sends some message $m$ at time $t$, all correct processes will receive + $m$ before $max\{t, GST\} + \Delta$. Furthermore, if a correct process $p$ + receives some message $m$ at time $t$, all correct processes will receive + $m$ before $max\{t, GST\} + \Delta$. \end{itemize} + + +The bound $\Delta$ and GST are system +parameters whose values are not required to be known for the safety of our +algorithm. Termination of the algorithm is guaranteed within a bounded duration +after GST. In practice, the algorithm will work correctly in the slightly +weaker variant of the model where the system alternates between (long enough) +good periods (corresponds to the \emph{after} GST period where system is +reliable and $\Delta$-timely) and bad periods (corresponds to the period +\emph{before} GST during which the system is asynchronous and messages can be +lost), but consideration of the GST model simplifies the discussion. + +We assume that process steps (which might include sending and receiving +messages) take zero time. Processes are equipped with clocks so they can +measure local timeouts. +Spoofing/impersonation attacks are assumed to be impossible at all times due to +the use of public-key cryptography, i.e., we assume that all protocol messages contains a digital signature. +Therefore, when a correct +process $q$ receives a signed message $m$ from its peer, the process $q$ can +verify who was the original sender of the message $m$ and if the message signature is valid. +We do not explicitly state a signature verification step in the pseudo-code of the algorithm to improve readability; +we assume that only messages with the valid signature are considered at that level (and messages with invalid signatures +are dropped). + + + +%Messages that are being gossiped are created by the consensus layer. We can + %think about consensus protocol as a content creator, which %defines what + %messages should be disseminated using the gossip protocol. A correct + %process creates the message for dissemination either i) %explicitly, by + %invoking \emph{send} function as part of the consensus protocol or ii) + %implicitly, by receiving a message from some other %process. Note that in + %the case ii) gossiping of messages is implicit, i.e., it happens without + %explicit send clause in the consensus algorithm %whenever a correct + %process receives some messages in the consensus algorithm\footnote{If a + %message is received by a correct process at %the consensus level then it + %is considered valid from the protocol point of view, i.e., it has a + %correct signature, a proper message structure %and a valid height and + %round number.}. + +%\item Processes keep resending messages (in case of failures or message loss) + %until all its peers get them. This ensures that every message %sent or + %received by a correct process is eventually received by all correct + %processes. + +\subsection{State Machine Replication} + +State machine replication (SMR) is a general approach for replicating services +modeled as a deterministic state machine~\cite{Lam78:cacm,Sch90:survey}. The +key idea of this approach is to guarantee that all replicas start in the same +state and then apply requests from clients in the same order, thereby +guaranteeing that the replicas' states will not diverge. Following +Schneider~\cite{Sch90:survey}, we note that the following is key for +implementing a replicated state machine tolerant to (Byzantine) faults: + +\begin{itemize} \item \emph{Replica Coordination.} All [non-faulty] replicas + receive and process the same sequence of requests. \end{itemize} + +Moreover, as Schneider also notes, this property can be decomposed into two +parts, \emph{Agreement} and \emph{Order}: Agreement requires all (non-faulty) +replicas to receive all requests, and Order requires that the order of received +requests is the same at all replicas. + +There is an additional requirement that needs to be ensured by Byzantine +tolerant state machine replication: only requests (called transactions in the +Tendermint terminology) proposed by clients are executed. In Tendermint, +transaction verification is the responsibility of the service that is being +replicated; upon receiving a transaction from the client, the Tendermint +process will ask the service if the request is valid, and only valid requests +will be processed. + + \subsection{Consensus} \label{sec:consensus} + +Tendermint solves state machine replication by sequentially executing consensus +instances to agree on each block of transactions that are +then executed by the service being replicated. We consider a variant of the +Byzantine consensus problem called Validity Predicate-based Byzantine consensus +that is motivated by blockchain systems~\cite{GLR17:red-belly-bc}. The problem +is defined by an agreement, a termination, and a validity property. + + \begin{itemize} \item \emph{Agreement:} No two correct processes decide on + different values. \item \emph{Termination:} All correct processes + eventually decide on a value. \item \emph{Validity:} A decided value + is valid, i.e., it satisfies the predefined predicate denoted + \emph{valid()}. \end{itemize} + + This variant of the Byzantine consensus problem has an application-specific + \emph{valid()} predicate to indicate whether a value is valid. In the context + of blockchain systems, for example, a value is not valid if it does not + contain an appropriate hash of the last value (block) added to the blockchain. diff --git a/cometbft/v0.38/spec/consensus/consensus-paper/homodel.sty b/cometbft/v0.38/spec/consensus/consensus-paper/homodel.sty new file mode 100644 index 00000000..19f83e92 --- /dev/null +++ b/cometbft/v0.38/spec/consensus/consensus-paper/homodel.sty @@ -0,0 +1,32 @@ +\newcommand{\NC}{\mbox{\it NC}} +\newcommand{\HO}{\mbox{\it HO}} +\newcommand{\AS}{\mbox{\it AS}} +\newcommand{\SK}{\mbox{\it SK}} +\newcommand{\SHO}{\mbox{\it SHO}} +\newcommand{\AHO}{\mbox{\it AHO}} +\newcommand{\CONS}{\mbox{\it CONS}} +\newcommand{\K}{\mbox{\it K}} + +\newcommand{\Alg}{\mathcal{A}} +\newcommand{\Pred}{\mathcal{P}} +\newcommand{\Spr}{S_p^r} +\newcommand{\Tpr}{T_p^r} +\newcommand{\mupr}{\vec{\mu}_p^{\,r}} + +\newcommand{\MSpr}{S_p^{\rho}} +\newcommand{\MTpr}{T_p^{\rho}} + + + +\newconstruct{\SEND}{$\Spr$:}{}{\ENDSEND}{} +\newconstruct{\TRAN}{$\Tpr$:}{}{\ENDTRAN}{} +\newconstruct{\ROUND}{\textbf{Round}}{\!\textbf{:}}{\ENDROUND}{} +\newconstruct{\VARIABLES}{\textbf{Variables:}}{}{\ENDVARIABLES}{} +\newconstruct{\INIT}{\textbf{Initialization:}}{}{\ENDINIT}{} + +\newconstruct{\MSEND}{$\MSpr$:}{}{\ENDMSEND}{} +\newconstruct{\MTRAN}{$\MTpr$:}{}{\ENDMTRAN}{} + +\newconstruct{\SROUND}{\textbf{Selection Round}}{\!\textbf{:}}{\ENDSROUND}{} +\newconstruct{\VROUND}{\textbf{Validation Round}}{\!\textbf{:}}{\ENDVROUND}{} +\newconstruct{\DROUND}{\textbf{Decision Round}}{\!\textbf{:}}{\ENDDROUND}{} diff --git a/cometbft/v0.38/spec/consensus/consensus-paper/intro.tex b/cometbft/v0.38/spec/consensus/consensus-paper/intro.tex new file mode 100644 index 00000000..493b509e --- /dev/null +++ b/cometbft/v0.38/spec/consensus/consensus-paper/intro.tex @@ -0,0 +1,138 @@ +\section{Introduction} \label{sec:tendermint} + +Consensus is a fundamental problem in distributed computing. It +is important because of it's role in State Machine Replication (SMR), a generic +approach for replicating services that can be modeled as a deterministic state +machine~\cite{Lam78:cacm, Sch90:survey}. The key idea of this approach is that +service replicas start in the same initial state, and then execute requests +(also called transactions) in the same order; thereby guaranteeing that +replicas stay in sync with each other. The role of consensus in the SMR +approach is ensuring that all replicas receive transactions in the same order. +Traditionally, deployments of SMR based systems are in data-center settings +(local area network), have a small number of replicas (three to seven) and are +typically part of a single administration domain (e.g., Chubby +\cite{Bur:osdi06}); therefore they handle benign (crash) failures only, as more +general forms of failure (in particular, malicious or Byzantine faults) are +considered to occur with only negligible probability. + +The success of cryptocurrencies and blockchain systems in recent years (e.g., +\cite{Nak2012:bitcoin, But2014:ethereum}) pose a whole new set of challenges on +the design and deployment of SMR based systems: reaching agreement over wide +area network, among large number of nodes (hundreds or thousands) that are not +part of the same administrative domain, and where a subset of nodes can behave +maliciously (Byzantine faults). Furthermore, contrary to the previous +data-center deployments where nodes are fully connected to each other, in +blockchain systems, a node is only connected to a subset of other nodes, so +communication is achieved by gossip-based peer-to-peer protocols. +The new requirements demand designs and algorithms that are not necessarily +present in the classical academic literature on Byzantine fault tolerant +consensus (or SMR) systems (e.g., \cite{DLS88:jacm, CL02:tcs}) as the primary +focus was different setup. + +In this paper we describe a novel Byzantine-fault tolerant consensus algorithm +that is the core of the BFT SMR platform called Tendermint\footnote{The + Tendermint platform is available open source at + https://github.com/tendermint/tendermint.}. The Tendermint platform consists of +a high-performance BFT SMR implementation written in Go, a flexible interface +for +building arbitrary deterministic applications above the consensus, and a suite +of tools for deployment and management. + +The Tendermint consensus algorithm is inspired by the PBFT SMR +algorithm~\cite{CL99:osdi} and the DLS algorithm for authenticated faults (the +Algorithm 2 from \cite{DLS88:jacm}). Similar to DLS algorithm, Tendermint +proceeds in +rounds\footnote{Tendermint is not presented in the basic round model of + \cite{DLS88:jacm}. Furthermore, we use the term round differently than in + \cite{DLS88:jacm}; in Tendermint a round denotes a sequence of communication + steps instead of a single communication step in \cite{DLS88:jacm}.}, where each +round has a dedicated proposer (also called coordinator or +leader) and a process proceeds to a new round as part of normal +processing (not only in case the proposer is faulty or suspected as being faulty +by enough processes as in PBFT). +The communication pattern of each round is very similar to the "normal" case +of PBFT. Therefore, in preferable conditions (correct proposer, timely and +reliable communication between correct processes), Tendermint decides in three +communication steps (the same as PBFT). + +The major novelty and contribution of the Tendermint consensus algorithm is a +new termination mechanism. As explained in \cite{MHS09:opodis, RMS10:dsn}, the +existing BFT consensus (and SMR) algorithms for the partially synchronous +system model (for example PBFT~\cite{CL99:osdi}, \cite{DLS88:jacm}, +\cite{MA06:tdsc}) typically relies on the communication pattern illustrated in +Figure~\ref{ch3:fig:coordinator-change} for termination. The +Figure~\ref{ch3:fig:coordinator-change} illustrates messages exchanged during +the proposer change when processes start a new round\footnote{There is no + consistent terminology in the distributed computing terminology on naming + sequence of communication steps that corresponds to a logical unit. It is + sometimes called a round, phase or a view.}. It guarantees that eventually (ie. +after some Global Stabilization Time, GST), there exists a round with a correct +proposer that will bring the system into a univalent configuration. +Intuitively, in a round in which the proposed value is accepted +by all correct processes, and communication between correct processes is +timely and reliable, all correct processes decide. + + +\begin{figure}[tbh!] \def\rdstretch{5} \def\ystretch{3} \centering + \begin{rounddiag}{4}{2} \round{1}{~} \rdmessage{1}{1}{$v_1$} + \rdmessage{2}{1}{$v_2$} \rdmessage{3}{1}{$v_3$} \rdmessage{4}{1}{$v_4$} + \round{2}{~} \rdmessage{1}{1}{$x, [v_{1..4}]$} + \rdmessage{1}{2}{$~~~~~~x, [v_{1..4}]$} \rdmessage{1}{3}{$~~~~~~~~x, + [v_{1..4}]$} \rdmessage{1}{4}{$~~~~~~~x, [v_{1..4}]$} \end{rounddiag} + \vspace{-5mm} \caption{\boldmath Proposer (coordinator) change: $p_1$ is the + new proposer.} \label{ch3:fig:coordinator-change} \end{figure} + +To ensure that a proposed value is accepted by all correct +processes\footnote{The proposed value is not blindly accepted by correct + processes in BFT algorithms. A correct process always verifies if the proposed + value is safe to be accepted so that safety properties of consensus are not + violated.} +a proposer will 1) build the global state by receiving messages from other +processes, 2) select the safe value to propose and 3) send the selected value +together with the signed messages +received in the first step to support it. The +value $v_i$ that a correct process sends to the next proposer normally +corresponds to a value the process considers as acceptable for a decision: + +\begin{itemize} \item in PBFT~\cite{CL99:osdi} and DLS~\cite{DLS88:jacm} it is + not the value itself but a set of $2f+1$ signed messages with the same + value id, \item in Fast Byzantine Paxos~\cite{MA06:tdsc} the value + itself is being sent. \end{itemize} + +In both cases, using this mechanism in our system model (ie. high +number of nodes over gossip based network) would have high communication +complexity that increases with the number of processes: in the first case as +the message sent depends on the total number of processes, and in the second +case as the value (block of transactions) is sent by each process. The set of +messages received in the first step are normally piggybacked on the proposal +message (in the Figure~\ref{ch3:fig:coordinator-change} denoted with +$[v_{1..4}]$) to justify the choice of the selected value $x$. Note that +sending this message also does not scale with the number of processes in the +system. + +We designed a novel termination mechanism for Tendermint that better suits the +system model we consider. It does not require additional communication (neither +sending new messages nor piggybacking information on the existing messages) and +it is fully based on the communication pattern that is very similar to the +normal case in PBFT \cite{CL99:osdi}. Therefore, there is only a single mode of +execution in Tendermint, i.e., there is no separation between the normal and +the recovery mode, which is the case in other PBFT-like protocols (e.g., +\cite{CL99:osdi}, \cite{Ver09:spinning} or \cite{Cle09:aardvark}). We believe +this makes Tendermint simpler to understand and implement correctly. + +Note that the orthogonal approach for reducing message complexity in order to +improve +scalability and decentralization (number of processes) of BFT consensus +algorithms is using advanced cryptography (for example Boneh-Lynn-Shacham (BLS) +signatures \cite{BLS2001:crypto}) as done for example in SBFT +\cite{Gue2018:sbft}. + +The remainder of the paper is as follows: Section~\ref{sec:definitions} defines +the system model and gives the problem definitions. Tendermint +consensus algorithm is presented in Section~\ref{sec:tendermint} and the +proofs are given in Section~\ref{sec:proof}. We conclude in +Section~\ref{sec:conclusion}. + + + + diff --git a/cometbft/v0.38/spec/consensus/consensus-paper/latex8.bst b/cometbft/v0.38/spec/consensus/consensus-paper/latex8.bst new file mode 100644 index 00000000..2c7af564 --- /dev/null +++ b/cometbft/v0.38/spec/consensus/consensus-paper/latex8.bst @@ -0,0 +1,1124 @@ + +% --------------------------------------------------------------- +% +% $Id: latex8.bst,v 1.1 1995/09/15 15:13:49 ienne Exp $ +% +% by Paolo.Ienne@di.epfl.ch +% + +% --------------------------------------------------------------- +% +% no guarantee is given that the format corresponds perfectly to +% IEEE 8.5" x 11" Proceedings, but most features should be ok. +% +% --------------------------------------------------------------- +% +% `latex8' from BibTeX standard bibliography style `abbrv' +% version 0.99a for BibTeX versions 0.99a or later, LaTeX version 2.09. +% Copyright (C) 1985, all rights reserved. +% Copying of this file is authorized only if either +% (1) you make absolutely no changes to your copy, including name, or +% (2) if you do make changes, you name it something other than +% btxbst.doc, plain.bst, unsrt.bst, alpha.bst, and abbrv.bst. +% This restriction helps ensure that all standard styles are identical. +% The file btxbst.doc has the documentation for this style. + +ENTRY + { address + author + booktitle + chapter + edition + editor + howpublished + institution + journal + key + month + note + number + organization + pages + publisher + school + series + title + type + volume + year + } + {} + { label } + +INTEGERS { output.state before.all mid.sentence after.sentence after.block } + +FUNCTION {init.state.consts} +{ #0 'before.all := + #1 'mid.sentence := + #2 'after.sentence := + #3 'after.block := +} + +STRINGS { s t } + +FUNCTION {output.nonnull} +{ 's := + output.state mid.sentence = + { ", " * write$ } + { output.state after.block = + { add.period$ write$ + newline$ + "\newblock " write$ + } + { output.state before.all = + 'write$ + { add.period$ " " * write$ } + if$ + } + if$ + mid.sentence 'output.state := + } + if$ + s +} + +FUNCTION {output} +{ duplicate$ empty$ + 'pop$ + 'output.nonnull + if$ +} + +FUNCTION {output.check} +{ 't := + duplicate$ empty$ + { pop$ "empty " t * " in " * cite$ * warning$ } + 'output.nonnull + if$ +} + +FUNCTION {output.bibitem} +{ newline$ + "\bibitem{" write$ + cite$ write$ + "}" write$ + newline$ + "" + before.all 'output.state := +} + +FUNCTION {fin.entry} +{ add.period$ + write$ + newline$ +} + +FUNCTION {new.block} +{ output.state before.all = + 'skip$ + { after.block 'output.state := } + if$ +} + +FUNCTION {new.sentence} +{ output.state after.block = + 'skip$ + { output.state before.all = + 'skip$ + { after.sentence 'output.state := } + if$ + } + if$ +} + +FUNCTION {not} +{ { #0 } + { #1 } + if$ +} + +FUNCTION {and} +{ 'skip$ + { pop$ #0 } + if$ +} + +FUNCTION {or} +{ { pop$ #1 } + 'skip$ + if$ +} + +FUNCTION {new.block.checka} +{ empty$ + 'skip$ + 'new.block + if$ +} + +FUNCTION {new.block.checkb} +{ empty$ + swap$ empty$ + and + 'skip$ + 'new.block + if$ +} + +FUNCTION {new.sentence.checka} +{ empty$ + 'skip$ + 'new.sentence + if$ +} + +FUNCTION {new.sentence.checkb} +{ empty$ + swap$ empty$ + and + 'skip$ + 'new.sentence + if$ +} + +FUNCTION {field.or.null} +{ duplicate$ empty$ + { pop$ "" } + 'skip$ + if$ +} + +FUNCTION {emphasize} +{ duplicate$ empty$ + { pop$ "" } + { "{\em " swap$ * "}" * } + if$ +} + +INTEGERS { nameptr namesleft numnames } + +FUNCTION {format.names} +{ 's := + #1 'nameptr := + s num.names$ 'numnames := + numnames 'namesleft := + { namesleft #0 > } + { s nameptr "{f.~}{vv~}{ll}{, jj}" format.name$ 't := + nameptr #1 > + { namesleft #1 > + { ", " * t * } + { numnames #2 > + { "," * } + 'skip$ + if$ + t "others" = + { " et~al." * } + { " and " * t * } + if$ + } + if$ + } + 't + if$ + nameptr #1 + 'nameptr := + + namesleft #1 - 'namesleft := + } + while$ +} + +FUNCTION {format.authors} +{ author empty$ + { "" } + { author format.names } + if$ +} + +FUNCTION {format.editors} +{ editor empty$ + { "" } + { editor format.names + editor num.names$ #1 > + { ", editors" * } + { ", editor" * } + if$ + } + if$ +} + +FUNCTION {format.title} +{ title empty$ + { "" } + { title "t" change.case$ } + if$ +} + +FUNCTION {n.dashify} +{ 't := + "" + { t empty$ not } + { t #1 #1 substring$ "-" = + { t #1 #2 substring$ "--" = not + { "--" * + t #2 global.max$ substring$ 't := + } + { { t #1 #1 substring$ "-" = } + { "-" * + t #2 global.max$ substring$ 't := + } + while$ + } + if$ + } + { t #1 #1 substring$ * + t #2 global.max$ substring$ 't := + } + if$ + } + while$ +} + +FUNCTION {format.date} +{ year empty$ + { month empty$ + { "" } + { "there's a month but no year in " cite$ * warning$ + month + } + if$ + } + { month empty$ + 'year + { month " " * year * } + if$ + } + if$ +} + +FUNCTION {format.btitle} +{ title emphasize +} + +FUNCTION {tie.or.space.connect} +{ duplicate$ text.length$ #3 < + { "~" } + { " " } + if$ + swap$ * * +} + +FUNCTION {either.or.check} +{ empty$ + 'pop$ + { "can't use both " swap$ * " fields in " * cite$ * warning$ } + if$ +} + +FUNCTION {format.bvolume} +{ volume empty$ + { "" } + { "volume" volume tie.or.space.connect + series empty$ + 'skip$ + { " of " * series emphasize * } + if$ + "volume and number" number either.or.check + } + if$ +} + +FUNCTION {format.number.series} +{ volume empty$ + { number empty$ + { series field.or.null } + { output.state mid.sentence = + { "number" } + { "Number" } + if$ + number tie.or.space.connect + series empty$ + { "there's a number but no series in " cite$ * warning$ } + { " in " * series * } + if$ + } + if$ + } + { "" } + if$ +} + +FUNCTION {format.edition} +{ edition empty$ + { "" } + { output.state mid.sentence = + { edition "l" change.case$ " edition" * } + { edition "t" change.case$ " edition" * } + if$ + } + if$ +} + +INTEGERS { multiresult } + +FUNCTION {multi.page.check} +{ 't := + #0 'multiresult := + { multiresult not + t empty$ not + and + } + { t #1 #1 substring$ + duplicate$ "-" = + swap$ duplicate$ "," = + swap$ "+" = + or or + { #1 'multiresult := } + { t #2 global.max$ substring$ 't := } + if$ + } + while$ + multiresult +} + +FUNCTION {format.pages} +{ pages empty$ + { "" } + { pages multi.page.check + { "pages" pages n.dashify tie.or.space.connect } + { "page" pages tie.or.space.connect } + if$ + } + if$ +} + +FUNCTION {format.vol.num.pages} +{ volume field.or.null + number empty$ + 'skip$ + { "(" number * ")" * * + volume empty$ + { "there's a number but no volume in " cite$ * warning$ } + 'skip$ + if$ + } + if$ + pages empty$ + 'skip$ + { duplicate$ empty$ + { pop$ format.pages } + { ":" * pages n.dashify * } + if$ + } + if$ +} + +FUNCTION {format.chapter.pages} +{ chapter empty$ + 'format.pages + { type empty$ + { "chapter" } + { type "l" change.case$ } + if$ + chapter tie.or.space.connect + pages empty$ + 'skip$ + { ", " * format.pages * } + if$ + } + if$ +} + +FUNCTION {format.in.ed.booktitle} +{ booktitle empty$ + { "" } + { editor empty$ + { "In " booktitle emphasize * } + { "In " format.editors * ", " * booktitle emphasize * } + if$ + } + if$ +} + +FUNCTION {empty.misc.check} + +{ author empty$ title empty$ howpublished empty$ + month empty$ year empty$ note empty$ + and and and and and + key empty$ not and + { "all relevant fields are empty in " cite$ * warning$ } + 'skip$ + if$ +} + +FUNCTION {format.thesis.type} +{ type empty$ + 'skip$ + { pop$ + type "t" change.case$ + } + if$ +} + +FUNCTION {format.tr.number} +{ type empty$ + { "Technical Report" } + 'type + if$ + number empty$ + { "t" change.case$ } + { number tie.or.space.connect } + if$ +} + +FUNCTION {format.article.crossref} +{ key empty$ + { journal empty$ + { "need key or journal for " cite$ * " to crossref " * crossref * + warning$ + "" + } + { "In {\em " journal * "\/}" * } + if$ + } + { "In " key * } + if$ + " \cite{" * crossref * "}" * +} + +FUNCTION {format.crossref.editor} +{ editor #1 "{vv~}{ll}" format.name$ + editor num.names$ duplicate$ + #2 > + { pop$ " et~al." * } + { #2 < + 'skip$ + { editor #2 "{ff }{vv }{ll}{ jj}" format.name$ "others" = + { " et~al." * } + { " and " * editor #2 "{vv~}{ll}" format.name$ * } + if$ + } + if$ + } + if$ +} + +FUNCTION {format.book.crossref} +{ volume empty$ + { "empty volume in " cite$ * "'s crossref of " * crossref * warning$ + "In " + } + { "Volume" volume tie.or.space.connect + " of " * + } + if$ + editor empty$ + editor field.or.null author field.or.null = + or + { key empty$ + { series empty$ + { "need editor, key, or series for " cite$ * " to crossref " * + crossref * warning$ + "" * + } + { "{\em " * series * "\/}" * } + if$ + } + { key * } + if$ + } + { format.crossref.editor * } + if$ + " \cite{" * crossref * "}" * +} + +FUNCTION {format.incoll.inproc.crossref} +{ editor empty$ + editor field.or.null author field.or.null = + or + { key empty$ + { booktitle empty$ + { "need editor, key, or booktitle for " cite$ * " to crossref " * + crossref * warning$ + "" + } + { "In {\em " booktitle * "\/}" * } + if$ + } + { "In " key * } + if$ + } + { "In " format.crossref.editor * } + if$ + " \cite{" * crossref * "}" * +} + +FUNCTION {article} +{ output.bibitem + format.authors "author" output.check + new.block + format.title "title" output.check + new.block + crossref missing$ + { journal emphasize "journal" output.check + format.vol.num.pages output + format.date "year" output.check + } + { format.article.crossref output.nonnull + format.pages output + } + if$ + new.block + note output + fin.entry +} + +FUNCTION {book} +{ output.bibitem + author empty$ + { format.editors "author and editor" output.check } + { format.authors output.nonnull + crossref missing$ + { "author and editor" editor either.or.check } + 'skip$ + if$ + } + if$ + new.block + format.btitle "title" output.check + crossref missing$ + { format.bvolume output + new.block + format.number.series output + new.sentence + publisher "publisher" output.check + address output + } + { new.block + format.book.crossref output.nonnull + } + if$ + format.edition output + format.date "year" output.check + new.block + note output + fin.entry +} + +FUNCTION {booklet} +{ output.bibitem + format.authors output + new.block + format.title "title" output.check + howpublished address new.block.checkb + howpublished output + address output + format.date output + new.block + note output + fin.entry +} + +FUNCTION {inbook} +{ output.bibitem + author empty$ + { format.editors "author and editor" output.check } + { format.authors output.nonnull + + crossref missing$ + { "author and editor" editor either.or.check } + 'skip$ + if$ + } + if$ + new.block + format.btitle "title" output.check + crossref missing$ + { format.bvolume output + format.chapter.pages "chapter and pages" output.check + new.block + format.number.series output + new.sentence + publisher "publisher" output.check + address output + } + { format.chapter.pages "chapter and pages" output.check + new.block + format.book.crossref output.nonnull + } + if$ + format.edition output + format.date "year" output.check + new.block + note output + fin.entry +} + +FUNCTION {incollection} +{ output.bibitem + format.authors "author" output.check + new.block + format.title "title" output.check + new.block + crossref missing$ + { format.in.ed.booktitle "booktitle" output.check + format.bvolume output + format.number.series output + format.chapter.pages output + new.sentence + publisher "publisher" output.check + address output + format.edition output + format.date "year" output.check + } + { format.incoll.inproc.crossref output.nonnull + format.chapter.pages output + } + if$ + new.block + note output + fin.entry +} + +FUNCTION {inproceedings} +{ output.bibitem + format.authors "author" output.check + new.block + format.title "title" output.check + new.block + crossref missing$ + { format.in.ed.booktitle "booktitle" output.check + format.bvolume output + format.number.series output + format.pages output + address empty$ + { organization publisher new.sentence.checkb + organization output + publisher output + format.date "year" output.check + } + { address output.nonnull + format.date "year" output.check + new.sentence + organization output + publisher output + } + if$ + } + { format.incoll.inproc.crossref output.nonnull + format.pages output + } + if$ + new.block + note output + fin.entry +} + +FUNCTION {conference} { inproceedings } + +FUNCTION {manual} +{ output.bibitem + author empty$ + { organization empty$ + 'skip$ + { organization output.nonnull + address output + } + if$ + } + { format.authors output.nonnull } + if$ + new.block + format.btitle "title" output.check + author empty$ + { organization empty$ + { address new.block.checka + address output + } + 'skip$ + if$ + } + { organization address new.block.checkb + organization output + address output + } + if$ + format.edition output + format.date output + new.block + note output + fin.entry +} + +FUNCTION {mastersthesis} +{ output.bibitem + format.authors "author" output.check + new.block + format.title "title" output.check + new.block + "Master's thesis" format.thesis.type output.nonnull + school "school" output.check + address output + format.date "year" output.check + new.block + note output + fin.entry +} + +FUNCTION {misc} +{ output.bibitem + format.authors output + title howpublished new.block.checkb + format.title output + howpublished new.block.checka + howpublished output + format.date output + new.block + note output + fin.entry + empty.misc.check +} + +FUNCTION {phdthesis} +{ output.bibitem + format.authors "author" output.check + new.block + format.btitle "title" output.check + new.block + "PhD thesis" format.thesis.type output.nonnull + school "school" output.check + address output + format.date "year" output.check + new.block + note output + fin.entry +} + +FUNCTION {proceedings} +{ output.bibitem + editor empty$ + { organization output } + { format.editors output.nonnull } + + if$ + new.block + format.btitle "title" output.check + format.bvolume output + format.number.series output + address empty$ + { editor empty$ + { publisher new.sentence.checka } + { organization publisher new.sentence.checkb + organization output + } + if$ + publisher output + format.date "year" output.check + } + { address output.nonnull + format.date "year" output.check + new.sentence + editor empty$ + 'skip$ + { organization output } + if$ + publisher output + } + if$ + new.block + note output + fin.entry +} + +FUNCTION {techreport} +{ output.bibitem + format.authors "author" output.check + new.block + format.title "title" output.check + new.block + format.tr.number output.nonnull + institution "institution" output.check + address output + format.date "year" output.check + new.block + note output + fin.entry +} + +FUNCTION {unpublished} +{ output.bibitem + format.authors "author" output.check + new.block + format.title "title" output.check + new.block + note "note" output.check + format.date output + fin.entry +} + +FUNCTION {default.type} { misc } + +MACRO {jan} {"Jan."} + +MACRO {feb} {"Feb."} + +MACRO {mar} {"Mar."} + +MACRO {apr} {"Apr."} + +MACRO {may} {"May"} + +MACRO {jun} {"June"} + +MACRO {jul} {"July"} + +MACRO {aug} {"Aug."} + +MACRO {sep} {"Sept."} + +MACRO {oct} {"Oct."} + +MACRO {nov} {"Nov."} + +MACRO {dec} {"Dec."} + +MACRO {acmcs} {"ACM Comput. Surv."} + +MACRO {acta} {"Acta Inf."} + +MACRO {cacm} {"Commun. ACM"} + +MACRO {ibmjrd} {"IBM J. Res. Dev."} + +MACRO {ibmsj} {"IBM Syst.~J."} + +MACRO {ieeese} {"IEEE Trans. Softw. Eng."} + +MACRO {ieeetc} {"IEEE Trans. Comput."} + +MACRO {ieeetcad} + {"IEEE Trans. Comput.-Aided Design Integrated Circuits"} + +MACRO {ipl} {"Inf. Process. Lett."} + +MACRO {jacm} {"J.~ACM"} + +MACRO {jcss} {"J.~Comput. Syst. Sci."} + +MACRO {scp} {"Sci. Comput. Programming"} + +MACRO {sicomp} {"SIAM J. Comput."} + +MACRO {tocs} {"ACM Trans. Comput. Syst."} + +MACRO {tods} {"ACM Trans. Database Syst."} + +MACRO {tog} {"ACM Trans. Gr."} + +MACRO {toms} {"ACM Trans. Math. Softw."} + +MACRO {toois} {"ACM Trans. Office Inf. Syst."} + +MACRO {toplas} {"ACM Trans. Prog. Lang. Syst."} + +MACRO {tcs} {"Theoretical Comput. Sci."} + +READ + +FUNCTION {sortify} +{ purify$ + "l" change.case$ +} + +INTEGERS { len } + +FUNCTION {chop.word} +{ 's := + 'len := + s #1 len substring$ = + { s len #1 + global.max$ substring$ } + 's + if$ +} + +FUNCTION {sort.format.names} +{ 's := + #1 'nameptr := + "" + s num.names$ 'numnames := + numnames 'namesleft := + { namesleft #0 > } + { nameptr #1 > + { " " * } + 'skip$ + if$ + s nameptr "{vv{ } }{ll{ }}{ f{ }}{ jj{ }}" format.name$ 't := + nameptr numnames = t "others" = and + { "et al" * } + { t sortify * } + if$ + nameptr #1 + 'nameptr := + namesleft #1 - 'namesleft := + } + while$ +} + +FUNCTION {sort.format.title} +{ 't := + "A " #2 + "An " #3 + "The " #4 t chop.word + chop.word + chop.word + sortify + #1 global.max$ substring$ +} + +FUNCTION {author.sort} +{ author empty$ + { key empty$ + { "to sort, need author or key in " cite$ * warning$ + "" + } + { key sortify } + if$ + } + { author sort.format.names } + if$ +} + +FUNCTION {author.editor.sort} +{ author empty$ + { editor empty$ + { key empty$ + { "to sort, need author, editor, or key in " cite$ * warning$ + "" + } + { key sortify } + if$ + } + { editor sort.format.names } + if$ + } + { author sort.format.names } + if$ +} + +FUNCTION {author.organization.sort} +{ author empty$ + + { organization empty$ + { key empty$ + { "to sort, need author, organization, or key in " cite$ * warning$ + "" + } + { key sortify } + if$ + } + { "The " #4 organization chop.word sortify } + if$ + } + { author sort.format.names } + if$ +} + +FUNCTION {editor.organization.sort} +{ editor empty$ + { organization empty$ + { key empty$ + { "to sort, need editor, organization, or key in " cite$ * warning$ + "" + } + { key sortify } + if$ + } + { "The " #4 organization chop.word sortify } + if$ + } + { editor sort.format.names } + if$ +} + +FUNCTION {presort} +{ type$ "book" = + type$ "inbook" = + or + 'author.editor.sort + { type$ "proceedings" = + 'editor.organization.sort + { type$ "manual" = + 'author.organization.sort + 'author.sort + if$ + } + if$ + } + if$ + " " + * + year field.or.null sortify + * + " " + * + title field.or.null + sort.format.title + * + #1 entry.max$ substring$ + 'sort.key$ := +} + +ITERATE {presort} + +SORT + +STRINGS { longest.label } + +INTEGERS { number.label longest.label.width } + +FUNCTION {initialize.longest.label} +{ "" 'longest.label := + #1 'number.label := + #0 'longest.label.width := +} + +FUNCTION {longest.label.pass} +{ number.label int.to.str$ 'label := + number.label #1 + 'number.label := + label width$ longest.label.width > + { label 'longest.label := + label width$ 'longest.label.width := + } + 'skip$ + if$ +} + +EXECUTE {initialize.longest.label} + +ITERATE {longest.label.pass} + +FUNCTION {begin.bib} +{ preamble$ empty$ + 'skip$ + { preamble$ write$ newline$ } + if$ + "\begin{thebibliography}{" longest.label * + "}\setlength{\itemsep}{-1ex}\small" * write$ newline$ +} + +EXECUTE {begin.bib} + +EXECUTE {init.state.consts} + +ITERATE {call.type$} + +FUNCTION {end.bib} +{ newline$ + "\end{thebibliography}" write$ newline$ +} + +EXECUTE {end.bib} + +% end of file latex8.bst +% --------------------------------------------------------------- + + + diff --git a/cometbft/v0.38/spec/consensus/consensus-paper/latex8.sty b/cometbft/v0.38/spec/consensus/consensus-paper/latex8.sty new file mode 100644 index 00000000..1e6b0dc7 --- /dev/null +++ b/cometbft/v0.38/spec/consensus/consensus-paper/latex8.sty @@ -0,0 +1,168 @@ +% --------------------------------------------------------------- +% +% $Id: latex8.sty,v 1.2 1995/09/15 15:31:13 ienne Exp $ +% +% by Paolo.Ienne@di.epfl.ch +% +% --------------------------------------------------------------- +% +% no guarantee is given that the format corresponds perfectly to +% IEEE 8.5" x 11" Proceedings, but most features should be ok. +% +% --------------------------------------------------------------- +% with LaTeX2e: +% ============= +% +% use as +% \documentclass[times,10pt,twocolumn]{article} +% \usepackage{latex8} +% \usepackage{times} +% +% --------------------------------------------------------------- + +% with LaTeX 2.09: +% ================ +% +% use as +% \documentstyle[times,art10,twocolumn,latex8]{article} +% +% --------------------------------------------------------------- +% with both versions: +% =================== +% +% specify \pagestyle{empty} to omit page numbers in the final +% version +% +% specify references as +% \bibliographystyle{latex8} +% \bibliography{...your files...} +% +% use Section{} and SubSection{} instead of standard section{} +% and subsection{} to obtain headings in the form +% "1.3. My heading" +% +% --------------------------------------------------------------- + +\typeout{IEEE 8.5 x 11-Inch Proceedings Style `latex8.sty'.} + +% ten point helvetica bold required for captions +% in some sites the name of the helvetica bold font may differ, +% change the name here: +\font\tenhv = phvb at 10pt +%\font\tenhv = phvb7t at 10pt + +% eleven point times bold required for second-order headings +% \font\elvbf = cmbx10 scaled 1100 +\font\elvbf = ptmb scaled 1100 + +% set dimensions of columns, gap between columns, and paragraph indent +\setlength{\textheight}{8.875in} +\setlength{\textwidth}{6.875in} +\setlength{\columnsep}{0.3125in} +\setlength{\topmargin}{0in} +\setlength{\headheight}{0in} +\setlength{\headsep}{0in} +\setlength{\parindent}{1pc} +\setlength{\oddsidemargin}{-.304in} +\setlength{\evensidemargin}{-.304in} + +% memento from size10.clo +% \normalsize{\@setfontsize\normalsize\@xpt\@xiipt} +% \small{\@setfontsize\small\@ixpt{11}} +% \footnotesize{\@setfontsize\footnotesize\@viiipt{9.5}} +% \scriptsize{\@setfontsize\scriptsize\@viipt\@viiipt} +% \tiny{\@setfontsize\tiny\@vpt\@vipt} +% \large{\@setfontsize\large\@xiipt{14}} +% \Large{\@setfontsize\Large\@xivpt{18}} +% \LARGE{\@setfontsize\LARGE\@xviipt{22}} +% \huge{\@setfontsize\huge\@xxpt{25}} +% \Huge{\@setfontsize\Huge\@xxvpt{30}} + +\def\@maketitle + { + \newpage + \null + \vskip .375in + \begin{center} + {\Large \bf \@title \par} + % additional two empty lines at the end of the title + \vspace*{24pt} + { + \large + \lineskip .5em + \begin{tabular}[t]{c} + \@author + \end{tabular} + \par + } + % additional small space at the end of the author name + \vskip .5em + { + \large + \begin{tabular}[t]{c} + \@affiliation + \end{tabular} + \par + \ifx \@empty \@email + \else + \begin{tabular}{r@{~}l} + E-mail: & {\tt \@email} + \end{tabular} + \par + \fi + } + % additional empty line at the end of the title block + \vspace*{12pt} + \end{center} + } + +\def\abstract + {% + \centerline{\large\bf Abstract}% + \vspace*{12pt}% + \it% + } + +\def\endabstract + { + % additional empty line at the end of the abstract + \vspace*{12pt} + } + +\def\affiliation#1{\gdef\@affiliation{#1}} \gdef\@affiliation{} + +\def\email#1{\gdef\@email{#1}} +\gdef\@email{} + +\newlength{\@ctmp} +\newlength{\@figindent} +\setlength{\@figindent}{1pc} + +\long\def\@makecaption#1#2{ + \vskip 10pt + \setbox\@tempboxa\hbox{\tenhv\noindent #1.~#2} + \setlength{\@ctmp}{\hsize} + \addtolength{\@ctmp}{-\@figindent}\addtolength{\@ctmp}{-\@figindent} + % IF longer than one indented paragraph line + \ifdim \wd\@tempboxa >\@ctmp + % THEN set as an indented paragraph + \begin{list}{}{\leftmargin\@figindent \rightmargin\leftmargin} + \item[]\tenhv #1.~#2\par + \end{list} + \else + % ELSE center + \hbox to\hsize{\hfil\box\@tempboxa\hfil} + \fi} + +% correct heading spacing and type +\def\section{\@startsection {section}{1}{\z@} + {14pt plus 2pt minus 2pt}{14pt plus 2pt minus 2pt} {\large\bf}} +\def\subsection{\@startsection {subsection}{2}{\z@} + {13pt plus 2pt minus 2pt}{13pt plus 2pt minus 2pt} {\elvbf}} + +% add the period after section numbers +\newcommand{\Section}[1]{\section{\hskip -1em.~#1}} +\newcommand{\SubSection}[1]{\subsection{\hskip -1em.~#1}} + +% end of file latex8.sty +% --------------------------------------------------------------- diff --git a/cometbft/v0.38/spec/consensus/consensus-paper/lit.bib b/cometbft/v0.38/spec/consensus/consensus-paper/lit.bib new file mode 100644 index 00000000..4abc83e7 --- /dev/null +++ b/cometbft/v0.38/spec/consensus/consensus-paper/lit.bib @@ -0,0 +1,1659 @@ +%--- conferences -------------------------------------------------- +@STRING{WDAG96 = "Proceedings of the 10th International Workshop + on Distributed Algorithms (WDAG'96)"} +@STRING{WDAG97 = "Proceedings of the 11th International Workshop + on Distributed Algorithms (WDAG'97)"} +@STRING{DISC98 = "Proceedings of the 12th International Conference + on Distributed Computing ({DISC}'98)"} +@STRING{DISC99 = "Proceedings of the 13th International Conference + on Distributed Computing ({DISC}'99)"} +@STRING{DISC98 = "Proceedings of the 13th International Conference + on Distributed Computing ({DISC}'98)"} +@STRING{DISC99 = "Proceedings of the 13th International Conference + on Distributed Computing ({DISC}'99)"} +@STRING{DISC00 = "Proceedings of the 14th International Conference + on Distributed Computing ({DISC}'00)"} +@STRING{DISC01 = "Proceedings of the 15th International Conference + on Distributed Computing ({DISC}'01)"} +@STRING{DISC02 = "Proceedings of the 16th International Conference + on Distributed Computing ({DISC}'02)"} +@STRING{DISC03 = "Proceedings of the 17th International Conference + on Distributed Computing ({DISC}'03)"} +@STRING{DISC04 = "Proceedings of the 18th International Conference + on Distributed Computing ({DISC}'04)"} +@STRING{DISC05 = "Proceedings of the 19th International Conference + on Distributed Computing ({DISC}'05)"} +@STRING{PODC83 = "Proceeding of the 1st Annual {ACM} Symposium on + Principles of Distributed Computing ({PODC}'83)"} +@STRING{PODC91 = "Proceeding of the 9th Annual {ACM} Symposium on + Principles of Distributed Computing ({PODC}'91)"} +@STRING{PODC94 = "Proceeding of the 12th Annual {ACM} Symposium on + Principles of Distributed Computing ({PODC}'94)"} +@STRING{PODC95 = "Proceeding of the 13th Annual {ACM} Symposium on + Principles of Distributed Computing ({PODC}'95)"} +@STRING{PODC96 = "Proceeding of the 14th Annual {ACM} Symposium on + Principles of Distributed Computing ({PODC}'96)"} +@STRING{PODC97 = "Proceeding of the 15th Annual {ACM} Symposium on + Principles of Distributed Computing ({PODC}'97)"} +@STRING{PODC98 = "Proceeding of the 16th Annual {ACM} Symposium on + Principles of Distributed Computing ({PODC}'98)"} +@STRING{PODC99 = "Proceeding of the 17th Annual {ACM} Symposium on + Principles of Distributed Computing ({PODC}'99)"} +@STRING{PODC00 = "Proceeding of the 18th Annual {ACM} Symposium on + Principles of Distributed Computing ({PODC}'00)"} +@STRING{PODC01 = "Proceeding of the 19th Annual {ACM} Symposium on + Principles of Distributed Computing ({PODC}'01)"} +@STRING{PODC02 = "Proceeding of the 20th Annual {ACM} Symposium on + Principles of Distributed Computing ({PODC}'02)"} +@STRING{PODC03 = "Proceeding of the 21st Annual {ACM} Symposium on + Principles of Distributed Computing ({PODC}'03)"} +@STRING{PODC03 = "Proceeding of the 22nd Annual {ACM} Symposium on + Principles of Distributed Computing ({PODC}'03)"} +@STRING{PODC04 = "Proceeding of the 23rd Annual {ACM} Symposium on + Principles of Distributed Computing ({PODC}'04)"} +@STRING{PODC05 = "Proceeding of the 24th Annual {ACM} Symposium on + Principles of Distributed Computing ({PODC}'05)"} +@STRING{PODC06 = "Proceedings of the 25th Annual {ACM} Symposium on + Principles of Distributed Computing ({PODC}'06)"} +@STRING{PODC07 = "Proceedings of the 26th Annual {ACM} Symposium on + Principles of Distributed Computing ({PODC}'07)"} +@STRING{STOC91 = "Proceedings of the 23rd Annual {ACM} Symposium on + Theory of Computing ({STOC}'91)"} +@STRING{WSS01 = "Proceedings of the 5th International Workshop on + Self-Stabilizing Systems ({WSS} '01)"} +@STRING{SSS06 = "Proceedings of the 8th International Symposium on + Stabilization, Safety, and Security of Distributed + Systems ({SSS} '06)"} +@STRING{DSN00 = "Dependable Systems and Networks ({DSN} 2000)"} +@STRING{DSN05 = "Dependable Systems and Networks ({DSN} 2005)"} +@STRING{DSN06 = "Dependable Systems and Networks ({DSN} 2006)"} +@STRING{DSN07 = "Dependable Systems and Networks ({DSN} 2007)"} + +%--- journals ----------------------------------------------------- +@STRING{PPL = "Parallel Processing Letters"} +@STRING{IPL = "Information Processing Letters"} +@STRING{DC = "Distributed Computing"} +@STRING{JACM = "Journal of the ACM"} +@STRING{IC = "Information and Control"} +@STRING{TCS = "Theoretical Computer Science"} +@STRING{ACMTCS = "ACM Transactions on Computer Systems"} +@STRING{TDSC = "Transactions on Dependable and Secure Computing"} +@STRING{TPLS = "ACM Trans. Program. Lang. Syst."} + +%--- publisher ---------------------------------------------------- +@STRING{ACM = "ACM Press"} +@STRING{IEEE = "IEEE"} +@STRING{SPR = "Springer-Verlag"} + +%--- institution -------------------------------------------------- +@STRING{TUAuto = {Technische Universit\"at Wien, Department of + Automation}} +@STRING{TUECS = {Technische Universit\"at Wien, Embedded Computing + Systems Group}} + + +%------------------------------------------------------------------ +@article{ABND+90:jacm, + author = {Hagit Attiya and Amotz Bar-Noy and Danny Dolev and + David Peleg and R{\"u}diger Reischuk}, + title = {Renaming in an asynchronous environment}, + journal = JACM, + volume = {37}, + number = {3}, + year = {1990}, + pages = {524--548}, + publisher = ACM, + address = {New York, NY, USA}, +} + +@article{ABND95:jacm, + author = {Hagit Attiya and Amotz Bar-Noy and Danny Dolev}, + title = {Sharing memory robustly in message-passing systems}, + journal = JACM, + volume = {42}, + number = {1}, + year = {1995}, + pages = {124--142}, + publisher = ACM, + address = {New York, NY, USA}, +} + +@inproceedings{ACKM04:podc, + author = {Ittai Abraham and Gregory Chockler and Idit Keidar + and Dahlia Malkhi}, + title = {Byzantine disk paxos: optimal resilience with + byzantine shared memory.}, + booktitle = PODC04, + year = {2004}, + pages = {226-235} +} + +@article{ACKM05:dc, + author = {Ittai Abraham and Gregory Chockler and Idit Keidar + and Dahlia Malkhi}, + title = {Byzantine disk paxos: optimal resilience with + byzantine shared memory.}, + journal = DC, + volume = {18}, + number = {5}, + year = {2006}, + pages = {387-408} +} + +@article{ACT00:dc, + author = "Marcos Kawazoe Aguilera and Wei Chen and Sam Toueg", + title = "Failure Detection and Consensus in the + Crash-Recovery Model", + journal = DC, + year = 2000, + month = apr, + volume = 13, + number = 2, + pages = "99--125", + url = + "http://www.cs.cornell.edu/home/sam/FDpapers/crash-recovery-finaldcversion.ps" +} + +@article{ACT00:siam, + author = "Marcos Kawazoe Aguilera and Wei Chen and Sam Toueg", + title = "On quiescent reliable communication", + journal = "SIAM Journal of Computing", + year = 2000, + volume = 29, + number = 6, + pages = "2040--2073", + month = apr +} + +@inproceedings{ACT97:wdag, + author = "Marcos Kawazoe Aguilera and Wei Chen and Sam Toueg", + title = "Heartbeat: A Timeout-Free Failure Detector for + Quiescent Reliable Communication", + booktitle = WDAG97, + year = 1997, + pages = "126--140", + url = + "http://simon.cs.cornell.edu/Info/People/weichen/research/mypapers/wdag97final.ps" +} + +@article{ACT98:disc, + author = "Marcos Kawazoe Aguilera and Wei Chen and Sam Toueg", + title = "Failure Detection and Consensus in the + Crash-Recovery Model", + journal = DISC98, + year = 1998, + pages = "231--245", + publisher = SPR +} + +@article{ACT99:tcs, + author = "Marcos Kawazoe Aguilera and Wei Chen and Sam Toueg", + title = "Using the Heartbeat Failure Detector for Quiescent + Reliable Communication and Consensus in + Partitionable Networks", + journal = "Theoretical Computer Science", + year = 1999, + month = jun, + volume = 220, + number = 1, + pages = "3--30", + url = + "http://www.cs.cornell.edu/home/sam/FDpapers/TCS98final.ps" +} + +@inproceedings{ADGF+04:ispdc, + author = {Anceaume, Emmanuelle and Delporte-Gallet, Carole and + Fauconnier, Hugues and Hurfin, Michel and Le Lann, + G{\'e}rard }, + title = {Designing Modular Services in the Scattered + Byzantine Failure Model.}, + booktitle = {ISPDC/HeteroPar}, + year = {2004}, + pages = {262-269} +} + +@inproceedings{ADGF+06:dsn, + author = {Marcos Kawazoe Aguilera and Carole Delporte-Gallet + and Hugues Fauconnier and Sam Toueg}, + title = {Consensus with Byzantine Failures and Little System + Synchrony.}, + booktitle = DSN06, + year = {2006}, + pages = {147-155} +} + +@inproceedings{ADGFT01:disc, + author = "Marcos Kawazoe Aguilera and Carole Delporte-Gallet + and Hugues Fauconnier and Sam Toueg", + title = "Stable Leader Election", + booktitle = DISC01, + year = 2001, + pages = "108--122", + publisher = SPR +} + +@inproceedings{ADGFT03:podc, + author = "Marcos K. Aguilera and Carole Delporte-Gallet and + Hugues Fauconnier and Sam Toueg", + title = "On implementing {O}mega with weak reliability and + synchrony assumptions", + booktitle = PODC03, + year = 2003, + publisher = ACM +} + +@inproceedings{ADGFT04:podc, + author = {Marcos K. Aguilera and Carole Delporte-Gallet and + Hugues Fauconnier and Sam Toueg}, + title = {Communication-efficient leader election and + consensus with limited link synchrony}, + booktitle = PODC04, + year = 2004, + pages = {328--337}, + address = {St. John's, Newfoundland, Canada}, + publisher = ACM +} + +@inproceedings{ADGFT06:dsn, + author = {Marcos Kawazoe Aguilera and Carole Delporte-Gallet + and Hugues Fauconnier and Sam Toueg}, + title = {Consensus with Byzantine Failures and Little System + Synchrony.}, + booktitle = DSN06, + year = 2006, + pages = {147-155}, + ee = + {http://doi.ieeecomputersociety.org/10.1109/DSN.2006.22}, + bibsource = {DBLP, http://dblp.uni-trier.de} +} + +@inproceedings{ADLS91:stoc, + author = "Hagit Attiya and Cynthia Dwork and Nancy A. Lynch + and Larry J. Stockmeyer", + title = "Bounds on the Time to Reach Agreement in the + Presence of Timing Uncertainty", + booktitle = STOC91, + year = 1991, + pages = "359--369", +} + +@article{AT99:ipl, + author = "Marcos Kawazoe Aguilera and Sam Toueg", + title = "A Simple Bivalency Proof that t -Resilient Consensus + Requires t + 1 Rounds", + journal = IPL, + volume = "71", + number = "3-4", + pages = "155--158", + year = "1999" +} + +@Book{AW04:book, + author = {Attiya, Hagit and Welch, Jennifer}, + title = {Distributed Computing}, + publisher = {John Wiley {\&} Sons}, + edition = {2nd}, + year = {2004} +} + +@Book{AW98:book, + author = {Hagit Attiya and Jennifer Welch}, + title = {Distributed Computing}, + publisher = {McGraw-Hill Publishing Company}, + year = {1998} +} + +@InBook{AW98:book:chap12, + author = {Hagit Attiya and Jennifer Welch}, + title = {Distributed Computing}, + publisher = {McGraw-Hill Publishing Company}, + year = {1998}, + chapter = {12, "Improving the fault-tolerance of algorithms"} +} + +@inproceedings{ABHMS11:disc, + author = {Hagit Attiya and + Fatemeh Borran and + Martin Hutle and + Zarko Milosevic and + Andr{\'e} Schiper}, + title = {Structured Derivation of Semi-Synchronous Algorithms}, + booktitle = {DISC}, + year = {2011}, + pages = {374-388} +} + +@inproceedings{BCBG+07:podc, + author = {Martin Biely and Bernadette Charron-Bost and Antoine + Gaillard and Martin Hutle and Andr{\'e} Schiper and + Josef Widder}, + title = {Tolerating Corrupted Communication}, + publisher = ACM, + booktitle = PODC07, + year = {2007} +} + +@InProceedings{BCBT96:wdag, + author = {Anindya Basu and Bernadette Charron-Bost and Sam + Toueg}, + title = {Simulating Reliable Links with Unreliable Links in + the Presence of Process Crashes}, + pages = {105--122}, + booktitle = {WDAG 1996}, + editor = {Babao{\u g}lu, {\"O}zalp}, + year = {1996}, + month = {Oct}, + volume = {1151}, + ISBN = {3-540-61769-8}, + pubisher = {Springer}, + series = {Lecture Notes in Computer Science}, +} + +@article{BDFG03:sigact, + author = "R. Boichat and P. Dutta and S. Frolund and + R. Guerraoui", + title = "Reconstructing {P}axos", + journal = "ACM SIGACT News", + year = "2003", + volume = "34", + number = "1", + pages = "47-67" +} + +@unpublished{BHR+06:note, + author = "Martin Biely and Martin Hutle and Sergio Rajsbaum + and Ulrich Schmid and Corentin Travers and Josef + Widder", + title = "Discussion note on moving timely links", + note = "Unpublished", + month = apr, + year = 2006 +} + +@article{BHRT03:jda, + author = {Roberto Baldoni and Jean-Michel H{\'e}lary and + Michel Raynal and L{\'e}naick Tanguy}, + title = {Consensus in Byzantine asynchronous systems.}, + journal = {J. Discrete Algorithms}, + volume = {1}, + number = {2}, + year = {2003}, + pages = {185-210}, + ee = {http://dx.doi.org/10.1016/S1570-8667(03)00025-X}, + bibsource = {DBLP, http://dblp.uni-trier.de} +} + +@unpublished{BHSS08:tdsc, + author = {Fatemeh Borran and Martin Hutle and Nuno Santos and + Andr{\'e} Schiper}, + title = {Solving Consensus with Communication Predicates: + A~Quantitative Approach}, + note = {Under submission}, + year = {2008} +} + +@inproceedings{Ben83:podc, + author = {Michael Ben-Or}, + title = {Another Advantage of Free Choice: Completely + Asynchronous Agreement Protocols}, + booktitle = {PODC}, + year = {1983}, +} + +@inproceedings{Bra04:podc, + author = {Bracha, Gabriel}, + title = {An asynchronous [(n - 1)/3]-resilient consensus protocol}, + booktitle = {PODC '84: Proceedings of the third annual ACM symposium on Principles of distributed computing}, + year = {1984}, + isbn = {0-89791-143-1}, + pages = {154--162}, + location = {Vancouver, British Columbia, Canada}, + doi = {http://doi.acm.org/10.1145/800222.806743}, + publisher = {ACM}, + address = {New York, NY, USA}, + } + + +@inproceedings{CBGS00:dsn, + author = "Bernadette Charron-Bost and Rachid Guerraoui and + Andr{\'{e}} Schiper", + title = "Synchronous System and Perfect Failure Detector: + {S}olvability and efficiency issues", + booktitle = DSN00, + publisher = "{IEEE} Computer Society", + address = "New York, {USA}", + pages = "523--532", + year = "2000" +} + +@inproceedings{CBS06:prdc, + author = {Bernadette Charron-Bost and Andr{\'e} Schiper}, + title = {Improving Fast Paxos: being optimistic with no + overhead}, + booktitle = {Pacific Rim Dependable Computing, Proceedings}, + year = {2006} +} + +@article{CBS09, + author = {B. Charron-Bost and A. Schiper}, + title = {The {H}eard-{O}f model: computing in distributed systems with benign failures}, + journal ={Distributed Computing}, + number = {1}, + volume = {22}, + pages = {49-71}, + year ={2009} + } + + +@article{CBS07:sigact, + author = {Bernadette Charron-Bost and Andr\'{e} Schiper}, + title = {Harmful dogmas in fault tolerant distributed + computing}, + journal = {SIGACT News}, + volume = {38}, + number = {1}, + year = {2007}, + pages = {53--61}, +} + +@techreport{CBS07:tr, + author = {Charron-Bost, Bernadette and Schiper, Andr{\'{e}}}, + title = {The Heard-Of Model: Unifying all Benign Failures}, + institution = {EPFL}, + year = 2007, + OPTnumber = {LSR-REPORT-2006-004} +} + +@article{CELT00:jacm, + author = {Soma Chaudhuri and Maurice Erlihy and Nancy A. Lynch + and Mark R. Tuttle}, + title = {Tight bounds for k-set agreement}, + journal = JACM, + volume = {47}, + number = {5}, + year = {2000}, + pages = {912--943}, + publisher = ACM, + address = {New York, NY, USA}, +} + +@article{CF99:tpds, + author = "Flaviu Cristian and Christof Fetzer", + title = "The Timed Asynchronous Distributed System Model", + journal = "IEEE Transactions on Parallel and Distributed + Systems", + volume = "10", + number = "6", + pages = "642--657", + year = "1999" +} + +@article{CHT96:jacm, + author = "Tushar Deepak Chandra and Vassos Hadzilacos and Sam + Toueg", + title = "The Weakest Failure Detector for Solving Consensus", + journal = {JACM}, + year = {1996}, +} + +@article{CL02:tcs, + author = {Miguel Castro and Barbara Liskov}, + title = {Practical byzantine fault tolerance and proactive + recovery}, + journal = {ACMTCS}, + year = {2002}, +} + +@inproceedings{CL99:osdi, + author = {Miguel Castro and Barbara Liskov}, + title = {Practical byzantine fault tolerance and proactive + recovery}, + booktitle = {Proceedings of the 3rd Symposium on Operating + Systems Design and Implementation}, + year = {1999}, + month = feb +} + +@inproceedings{CT91:podc, + author = {Tushar Deepak Chandra and Sam Toueg}, + title = {Unreliable Failure Detectors for Asynchronous + Systems (Preliminary Version)}, + booktitle = PODC91, + year = {1991}, + pages = {325-340} +} + +@article{CT96:jacm1, + author = "Tushar Deepak Chandra and Sam Toueg", + title = "Unreliable Failure Detectors for Reliable + Distributed Systems", + journal = {JACM}, + year = {1996}, +} + +@inproceedings{CTA00:dsn, + author = "Wei Chen and Sam Toueg and Marcos Kawazoe Aguilera", + title = "On the Quality of Service of Failure Detectors", + booktitle = "Proceedings IEEE International Conference on + Dependable Systems and Networks (DSN / FTCS'30)", + address = "New York City, USA", + year = 2000 +} + +@TechReport{DFKM96:tr, + author = {Danny Dolev and Roy Friedman and Idit Keidar and + Dahlia Malkhi}, + title = {Failure detectors in omission failure environments}, + institution = {Department of Computer Science, Cornell University}, + year = {1996}, + type = {Technical Report}, + number = {96-1608} +} + +@inproceedings{DG02:podc, + author = {Partha Dutta and Rachid Guerraoui}, + title = {The inherent price of indulgence}, + booktitle = PODC02, + year = 2002, + pages = {88--97}, + location = {Monterey, California}, + publisher = ACM, + address = {New York, NY, USA}, +} + +@inproceedings{DGFG+04:podc, + author = {Carole Delporte-Gallet and Hugues Fauconnier and + Rachid Guerraoui and Vassos Hadzilacos and Petr + Kouznetsov and Sam Toueg}, + title = {The weakest failure detectors to solve certain + fundamental problems in distributed computing}, + booktitle = PODC04, + year = 2004, + pages = {338--346}, + location = {St. John's, Newfoundland, Canada}, + publisher = ACM, + address = {New York, NY, USA} +} + +@inproceedings{DGL05:dsn, + author = {Partha Dutta and Rachid Guerraoui and Leslie + Lamport}, + title = {How Fast Can Eventual Synchrony Lead to Consensus?}, + booktitle = {Proceedings of the 2005 International Conference on + Dependable Systems and Networks (DSN'05)}, + pages = {22--27}, + year = {2005}, + address = {Los Alamitos, CA, USA} +} + +@article{DLS88:jacm, + author = "Cynthia Dwork and Nancy Lynch and Larry Stockmeyer", + title = "Consensus in the Presence of Partial Synchrony", + journal = {JACM}, + year = {1988}, +} + +@article{DPLL00:tcs, + author = "De Prisco, Roberto and Butler Lampson and Nancy + Lynch", + title = "Revisiting the {PAXOS} algorithm", + journal = TCS, + volume = "243", + number = "1--2", + pages = "35--91", + year = "2000" +} + +@techreport{DS97:tr, + author = {A. Doudou and A. Schiper}, + title = {Muteness Failure Detectors for Consensus with + {B}yzantine Processes}, + institution = {EPFL, Dept d'Informatique}, + year = {1997}, + type = {TR}, + month = {October}, + number = {97/230}, +} + +@inproceedings{DS98:podc, + author = {A. Doudou and A. Schiper}, + title = {Muteness Detectors for Consensus with {B}yzantine + Processes ({B}rief {A}nnouncement)}, + booktitle = {PODC}, + month = jul, + year = {1998} +} + +@article{DSU04:survey, + author = {D{\'e}fago, Xavier and Schiper, Andr{\'e} and Urb\'{a}n, P{\'e}ter}, + title = {Total order broadcast and multicast algorithms: Taxonomy and survey}, + journal = {ACM Comput. Surv.}, + issue_date = {December 2004}, + volume = {36}, + number = {4}, + month = dec, + year = {2004}, + issn = {0360-0300}, + pages = {372--421}, + numpages = {50}, + publisher = {ACM}, + address = {New York, NY, USA}, + keywords = {Distributed systems, agreement problems, atomic broadcast, atomic multicast, classification, distributed algorithms, fault-tolerance, global ordering, group communication, message passing, survey, taxonomy, total ordering}, +} + +@article{DeCandia07:dynamo, + author = {DeCandia, Giuseppe and Hastorun, Deniz and Jampani, Madan and Kakulapati, Gunavardhan and Lakshman, Avinash and Pilchin, Alex and Sivasubramanian, Swaminathan and Vosshall, Peter and Vogels, Werner}, + title = {Dynamo: amazon's highly available key-value store}, + journal = {SIGOPS Oper. Syst. Rev.}, + issue_date = {December 2007}, + volume = {41}, + number = {6}, + month = oct, + year = {2007}, + issn = {0163-5980}, + pages = {205--220}, + numpages = {16}, + publisher = {ACM}, + address = {New York, NY, USA}, + keywords = {performance, reliability, scalability}, +} + + +@book{Dol00:book, + author = {Shlomi Dolev}, + title = {Self-Stabilization}, + publisher = {The MIT Press}, + year = {2000} +} + +@inproceedings{FC95:podc, + author = "Christof Fetzer and Flaviu Cristian", + title = "Lower Bounds for Convergence Function Based Clock + Synchronization", + booktitle = PODC95, + year = 1995, + pages = "137--143" +} + +@article{FLP85:jacm, + author = "Michael J. Fischer and Nancy A. Lynch and + M. S. Paterson", + title = "Impossibility of Distributed Consensus with one + Faulty Process", + journal = {JACM}, + year = {1985}, +} + +@article{FMR05:tdsc, + author = {Roy Friedman and Achour Most{\'e}faoui and Michel + Raynal}, + title = {Simple and Efficient Oracle-Based Consensus + Protocols for Asynchronous Byzantine Systems.}, + journal = TDSC, + volume = {2}, + number = {1}, + year = {2005}, + pages = {46-56}, + ee = {http://dx.doi.org/10.1109/TDSC.2005.13}, + bibsource = {DBLP, http://dblp.uni-trier.de} +} + +@inproceedings{FS04:podc, + author = "Christof Fetzer and Ulrich Schmid", + title = "Brief announcement: on the possibility of consensus + in asynchronous systems with finite average response + times.", + booktitle = PODC04, + year = 2004, + pages = 402 +} + +@InProceedings{GL00:disc, + author = {Eli Gafni and Lesli Lamport}, + title = {Disk Paxos}, + booktitle = DISC00, + pages = {330--344}, + year = {2000}, +} + +@Article{GL03:dc, + author = {Eli Gafni and Lesli Lamport}, + title = {Disk Paxos}, + journal = DC, + year = 2003, + volume = {16}, + number = {1}, + pages = {1--20} +} + +@inproceedings{GP01:wss, + author = "Felix C. G{\"a}rtner and Stefan Pleisch", + title = "({I}m)Possibilities of Predicate Detection in + Crash-Affected Systems", + booktitle = WSS01, + year = 2001, + pages = "98--113" +} + +@inproceedings{GP02:disc, + author = "Felix C. G{\"a}rtner and Stefan Pleisch", + title = "Failure Detection Sequencers: Necessary and + Sufficient Information about Failures to Solve + Predicate Detection", + booktitle = DISC02, + year = 2002, + pages = "280--294" +} + +@inproceedings{GS96:wdag, + author = {Rachid Guerraoui and Andr{\'e} Schiper}, + title = {{``Gamma-Accurate''} Failure Detectors}, + booktitle = WDAG96, + year = {1996}, + pages = {269--286}, + publisher = SPR, + address = {London, UK} +} + +@inproceedings{Gaf98:podc, + author = {Eli Gafni}, + title = {Round-by-round fault detectors (extended abstract): + unifying synchrony and asynchrony}, + booktitle = PODC98, + year = {1998}, + pages = {143--152}, + address = {Puerto Vallarta, Mexico}, + publisher = ACM +} + +@incollection{Gra78:book, + author = {Jim N. Gray}, + title = {Notes on data base operating systems}, + booktitle = {Operating Systems: An Advanced Course}, + chapter = {3.F}, + publisher = {Springer}, + year = {1978}, + editor = {R. Bayer, R.M. Graham, G. Seegm\"uller}, + volume = {60}, + series = {Lecture Notes in Computer Science}, + address = {New York}, + pages = {465}, +} + +@InProceedings{HMR98:srds, + author = {Hurfin, M. and Mostefaoui, A. and Raynal, M.}, + title = {Consensus in asynchronous systems where processes + can crash and recover}, + booktitle = {Seventeenth IEEE Symposium on Reliable Distributed + Systems, Proceedings. }, + pages = { 280--286}, + year = {1998}, + address = {West Lafayette, IN}, + month = oct, + organization = {IEEE} +} + +@inproceedings{HMSZ06:sss, + author = "Martin Hutle and Dahlia Malkhi and Ulrich Schmid and + Lidong Zhou", + title = "Brief Announcement: Chasing the Weakest System Model + for Implementing {$\Omega$} and Consensus", + booktitle = SSS06, + year = 2006 +} + +@incollection{HT93:ds, + author = {Hadzilacos, Vassos and Toueg, Sam}, + title = {Fault-tolerant broadcasts and related problems}, + booktitle = {Distributed systems (2nd Ed.)}, + editor = {Mullender, Sape}, + year = {1993}, + isbn = {0-201-62427-3}, + pages = {97--145}, + numpages = {49} +} + + +@inproceedings{HS06:opodis, + author = {Heinrich Moser and Ulrich Schmid}, + title = {Optimal Clock Synchronization Revisited: Upper and + Lower Bounds in Real-Time Systems}, + booktitle = { Principles of Distributed Systems}, + pages = {94--109}, + year = {2006}, + volume = {4305}, + series = {Lecture Notes in Computer Science}, + publisher = SPR +} + +@techreport{HS06:tr, + author = {Martin Hutle and Andr{\'e} Schiper}, + title = { Communication predicates: A high-level abstraction + for coping with transient and dynamic faults}, + institution = {EPFL}, + number = { LSR-REPORT-2006-006 }, + year = {2006} +} + +@inproceedings{HS07:dsn, + author = {Martin Hutle and Andr{\'e} Schiper}, + title = { Communication predicates: A high-level abstraction + for coping with transient and dynamic faults}, + year = 2007, + booktitle = DSN07, + publisher = IEEE, + location = {Edinburgh,UK}, + pages = {92--10}, + month = jun +} + +@article{Her91:tpls, + author = {Maurice Herlihy}, + title = {Wait-free synchronization}, + journal = TPLS, + volume = {13}, + number = {1}, + year = {1991}, + pages = {124--149}, + publisher = ACM, + address = {New York, NY, USA}, +} + +@article{Kot09:zyzzyva, + author = {Kotla, Ramakrishna and Alvisi, Lorenzo and Dahlin, Mike and Clement, Allen and Wong, Edmund}, + title = {Zyzzyva: Speculative Byzantine fault tolerance}, + journal = {ACM Trans. Comput. Syst.}, + issue_date = {December 2009}, + volume = {27}, + number = {4}, + month = jan, + year = {2010}, + issn = {0734-2071}, + pages = {7:1--7:39}, + articleno = {7}, + numpages = {39}, + publisher = {ACM}, + address = {New York, NY, USA}, + keywords = {Byzantine fault tolerance, output commit, replication, speculative execution}, +} + + +@inproceedings{KMMS97:opodis, + author = "Kim Potter Kihlstrom and Louise E. Moser and + P. M. Melliar-Smith", + title = "Solving Consensus in a Byzantine Environment Using + an Unreliable Fault Detector", + booktitle = "Proceedings of the International Conference on + Principles of Distributed Systems (OPODIS)", + year = 1997, + month = dec, + address = "Chantilly, France", + pages = "61--75" +} + +@inproceedings{KS06:podc, + author = {Idit Keidar and Alexander Shraer}, + title = {Timeliness, failure-detectors, and consensus + performance}, + booktitle = PODC06, + year = {2006}, + pages = {169--178}, + location = {Denver, Colorado, USA}, + publisher = {ACM Press}, + address = {New York, NY, USA}, +} + +@InProceedings{LFA99:disc, + author = {Mikel Larrea and Antonio Fern\'andez and Sergio + Ar\'evalo}, + title = {Efficient algorithms to implement unreliable failure + detectors in partially synchronous systems}, + year = 1999, + month = sep, + pages = {34-48}, + series = "LNCS 1693", + booktitle = DISC99, + publisher = SPR, + address = {Bratislava, Slovaquia}, +} + +@article{LL84:ic, + author = "Jennifer Lundelius and Nancy A. Lynch", + title = "An Upper and Lower Bound for Clock Synchronization", + journal = IC, + volume = 62, + number = {2/3}, + year = 1984, + pages = {190--204} +} + +@techreport{LLS03:tr, + title = {How to Implement a Timer-free Perfect Failure + Detector in Partially Synchronous Systems}, + author = {Le Lann, G\'erard and Schmid, Ulrich}, + institution = TUAuto, + number = "183/1-127", + month = jan, + year = 2003 +} + +@article{LSP82:tpls, + author = {Leslie Lamport and Robert Shostak and Marshall + Pease}, + title = {The {B}yzantine Generals Problem}, + journal = {ACM Trans. Program. Lang. Syst.}, + year = {1982}, +} + +@inproceedings{Lam01:podc, + author = {Butler Lampson}, + title = {The ABCD's of Paxos}, + booktitle = {PODC}, + year = {2001}, + +} + +@inproceedings{Lam03:fddc, + author = {Leslie Lamport}, + title = {Lower Bounds for Asynchronous Consensus}, + booktitle = {Future Directions in Distributed Computing}, + pages = {22--23}, + year = {2003}, + editor = {Andr{\'e} Schiper and Alex A. Shvartsman and Hakim + Weatherspoon and Ben Y. Zhao}, + number = {2584}, + series = {Lecture Notes in Computer Science}, + publisher = SPR +} + +@techreport{Lam04:tr, + author = {Leslie Lamport}, + title = {Lower Bounds for Asynchronous Consensus}, + institution = {Microsoft Research}, + year = {2004}, + number = {MSR-TR-2004-72} +} + +@techreport{Lam05:tr, + author = {Leslie Lamport}, + title = {Fast Paxos}, + institution = {Microsoft Research}, + year = {2005}, + number = {MSR-TR-2005-12} +} + +@techreport{Lam05:tr-33, + author = {Leslie Lamport}, + title = {Generalized Consensus and Paxos}, + institution = {Microsoft Research}, + year = {2005}, + number = {MSR-TR-2005-33} +} + +@Misc{Lam06:slides, + author = {Leslie Lamport}, + title = {Byzantine Paxos}, + howpublished = {Unpublished slides}, + year = {2006} +} + +@Article{Lam86:dc, + author = {Lesli Lamport}, + title = {On Interprocess Communication--Part I: Basic + Formalism, Part II: Algorithms}, + journal = DC, + year = 1986, + volume = 1, + number = 2, + pages = {77--101} +} + +@Article {Lam98:tcs, + author = {Leslie Lamport}, + title = {The part-time parliament}, + journal = ACMTCS, + year = 1998, + volume = 16, + number = 2, + month = may, + pages = {133-169}, +} + +@book{Lyn96:book, + author = {Nancy Lynch}, + title = {Distributed Algorithms}, + publisher = {Morgan Kaufman}, + year = {1996}, +} + +@inproceedings{MA05:dsn, + author = {Martin, J.-P. and Alvisi, L. }, + title = {Fast Byzantine consensus}, + booktitle = DSN05, + pages = {402--411}, + year = {2005}, + month = jun, + organization = {IEEE}, +} + +@article{MA06:tdsc, + author = {Martin, J.-P. and Alvisi, L. }, + title = {Fast {B}yzantine Consensus}, + journal = {TDSC}, + year = {2006}, +} + +@InProceedings{MOZ05:dsn, + author = {Dahlia Malkhi and Florin Oprea and Lidong Zhou}, + title = {{$\Omega$} Meets Paxos: Leader Election and + Stability without Eventual Timely Links}, + booktitle = DSN05, + year = {2005} +} + +@inproceedings{MR00:podc, + author = "Achour Most{\'e}faoui and Michel Raynal", + title = "k-set agreement with limited accuracy failure + detectors", + booktitle = PODC00, + year = 2000, + pages = {143--152}, + location = {Portland, Oregon, United States}, + publisher = ACM +} + +@article{MR01:ppl, + author = "Achour Most{\'e}faoui and Michel Raynal", + title = "Leader-Based Consensus", + journal = PPL, + volume = 11, + number = 1, + year = 2001, + pages = {95--107} +} + +@techreport{OGS97:tr, + author = "Rui Oliveira and Rachid Guerraoui and {Andr\'e} + Schiper", + title = "Consensus in the crash-recover model", + number = "TR-97/239", + year = "1997" +} + +@article{PSL80:jacm, + author = {M. Pease and R. Shostak and L. Lamport}, + title = {Reaching Agreement in the Presence of Faults}, + journal = JACM, + volume = {27}, + number = {2}, + year = {1980}, + pages = {228--234}, + publisher = ACM, + address = ACMADDR, +} + +@article{ST87:jacm, + author = "T. K. Srikanth and Sam Toueg", + title = "Optimal clock synchronization", + journal = JACM, + volume = 34, + number = 3, + year = 1987, + pages = "626--645" +} + +@article{ST87:dc, + author = {T. K. Srikanth and Sam Toueg,}, + title = {Simulating authenticated broadcasts to derive simple fault-tolerant algorithms}, + journal = DC, + volume = {2}, + number = {2}, + year = {1987}, + pages = {80-94} +} + + +@inproceedings{SW89:stacs, + author = {Santoro, Nicola and Widmayer, Peter}, + title = {Time is not a healer}, + booktitle = {Proc.\ 6th Annual Symposium on Theor.\ Aspects of + Computer Science (STACS'89)}, + publisher = "Springer-Verlag", + series = {LNCS}, + volume = "349", + address = "Paderborn, Germany", + pages = "304-313", + year = "1989", + month = feb, +} + +@inproceedings{SW90:sigal, + author = {Nicola Santoro and Peter Widmayer}, + title = {Distributed Function Evaluation in the Presence of + Transmission Faults.}, + booktitle = {SIGAL International Symposium on Algorithms}, + year = {1990}, + pages = {358-367} +} + +@inproceedings{SWR02:icdcs, + author = {Ulrich Schmid and Bettina Weiss and John Rushby}, + title = {Formally Verified Byzantine Agreement in Presence of + Link Faults}, + booktitle = "22nd International Conference on Distributed + Computing Systems (ICDCS'02)", + year = 2002, + month = jul # " 2-5, ", + pages = "608--616", + address = "Vienna, Austria", +} + +@incollection{Sch93a:mullender, + Author = {F. B. Schneider}, + Title = {What Good are Models and What Models are Good}, + BookTitle = {Distributed Systems}, + Year = {1993}, + Editor = {Sape Mullender}, + Publisher = {ACM Press}, + Pages = {169-197}, +} + +@article{VL96:ic, + author = {George Varghese and Nancy A. Lynch}, + title = {A Tradeoff Between Safety and Liveness for + Randomized Coordinated Attack.}, + journal = {Inf. Comput.}, + volume = {128}, + number = {1}, + year = 1996, + pages = {57--71} +} + +@inproceedings{WGWB07:dsn, + title = {Synchronous Consensus with Mortal Byzantines}, + author = {Josef Widder and Günther Gridling and Bettina Weiss + and Jean-Paul Blanquart}, + year = {2007}, + booktitle = DSN07, + publisher = IEEE +} + +@inproceedings{Wid03:disc, + author = {Josef Widder}, + title = {Booting clock Synchronization in Partially + Synchronous Systems}, + booktitle = DISC03, + year = {2003}, + pages = {121--135} +} + +@techreport{Zie04:tr, + author = {Piotr Zieli{\'n}ski}, + title = {Paxos at War}, + institution = {University of Cambridge}, + year = {2004}, + number = {UCAM-CL-TR-593}, +} + +@article{Lam78:cacm, + author = {Leslie Lamport}, + title = {Time, clocks, and the ordering of events in a + distributed system}, + journal = {Commun. ACM}, + year = {1978}, +} + +@Article{Gue06:cj, + author = {Guerraoui, R. and Raynal, M.}, + journal = {The {C}omputer {J}ournal}, + title = {The {A}lpha of {I}ndulgent {C}onsensus}, + year = {2006} +} + +@Article{Gue03:toc, + affiliation = {EPFL}, + author = {Guerraoui, Rachid and Raynal, Michel}, + journal = {{IEEE} {T}rans. on {C}omputers}, + title = {The {I}nformation {S}tructure of {I}ndulgent {C}onsensus}, + year = {2004}, +} + +@techreport{Cas00, + author = {Castro, Miguel}, + title = {Practical {B}yzantine Fault-Tolerance. {PhD} thesis}, + institution = {MIT}, + year = 2000, +} + +@inproceedings{SongRSD08:icdcn, + author = {Yee Jiun Song and + Robbert van Renesse and + Fred B. Schneider and + Danny Dolev}, + title = {The Building Blocks of Consensus}, + booktitle = {ICDCN}, + year = {2008}, +} + + +@inproceedings{BS09:icdcn, + author = {Borran, Fatemeh and Schiper, Andr{\'e}}, + + title = {A {L}eader-free {B}yzantine {C}onsensus {A}lgorithm}, + note = {To appear in ICDCN, 2010}, +} + + +@inproceedings{MHS09:opodis, + author = {Zarko Milosevic and Martin Hutle and Andr{\'e} + Schiper}, + title = {Unifying {B}yzantine Consensus Algorithms with {W}eak + {I}nteractive {C}onsistency}, + note = {To appear in OPODIS 2009}, +} + +@inproceedings{MRR:dsn02, + author = {Most\'{e}faoui, Achour and Rajsbaum, Sergio and Raynal, Michel}, + title = {A Versatile and Modular Consensus Protocol}, + booktitle = {DSN}, + year = {2002}, + } + +@article{MR98:dc, + author = {Dahlia Malkhi and + Michael K. Reiter}, + title = {Byzantine Quorum Systems}, + journal = {Distributed Computing}, + year = {1998}, +} + +@inproceedings{Rei:ccs94, + author = {Reiter, Michael K.}, + title = {Secure agreement protocols: reliable and atomic group multicast in rampart}, + booktitle = {CCS}, + year = {1994}, + pages = {68--80}, + numpages = {13} +} + + +@techreport{RMS09-tr, + author = {Olivier R\"utti and Zarko Milosevic and Andr\'e Schiper}, + title = {{G}eneric construction of consensus algorithm for benign and {B}yzantine faults}, + institution = {EPFL-IC}, + number = {LSR-REPORT-2009-005}, + year = 2009, +} + +@inproceedings{Li:srds07, + author = {Li, Harry C. and Clement, Allen and Aiyer, Amitanand S. and Alvisi, Lorenzo}, + title = {The Paxos Register}, + booktitle = {SRDS}, + year = {2007}, + } + + @article{Amir11:prime, + author = {Amir, Yair and Coan, Brian and Kirsch, Jonathan and Lane, John}, + title = {Prime: Byzantine Replication under Attack}, + journal = {IEEE Trans. Dependable Secur. Comput.}, + issue_date = {July 2011}, + volume = {8}, + number = {4}, + month = jul, + year = {2011}, + issn = {1545-5971}, + pages = {564--577}, + numpages = {14}, + publisher = {IEEE Computer Society Press}, + address = {Los Alamitos, CA, USA}, + keywords = {Performance under attack, Byzantine fault tolerance, replicated state machines, distributed systems.}, +} + +@inproceedings{Mao08:mencius, + author = {Mao, Yanhua and Junqueira, Flavio P. and Marzullo, Keith}, + title = {Mencius: building efficient replicated state machines for WANs}, + booktitle = {OSDI}, + year = {2008}, + pages = {369--384}, + numpages = {16} +} + +@article{Sch90:survey, + author = {Schneider, Fred B.}, + title = {Implementing fault-tolerant services using the state machine approach: a tutorial}, + journal = {ACM Comput. Surv.}, + volume = {22}, + number = {4}, + month = dec, + year = {1990} +} + + +@techreport{HT94:TR, + author = {Hadzilacos, Vassos and Toueg, Sam}, + title = {A Modular Approach to Fault-Tolerant Broadcasts and Related Problems}, + year = {1994}, + source = {http://www.ncstrl.org:8900/ncstrl/servlet/search?formname=detail\&id=oai%3Ancstrlh%3Acornellcs%3ACORNELLCS%3ATR94-1425}, + publisher = {Cornell University}, + address = {Ithaca, NY, USA}, +} + +@inproceedings{Ver09:spinning, + author = {Veronese, Giuliana Santos and Correia, Miguel and Bessani, Alysson Neves and Lung, Lau Cheuk}, + title = {Spin One's Wheels? Byzantine Fault Tolerance with a Spinning Primary}, + booktitle = {SRDS}, + year = {2009}, + numpages = {10} +} + +@inproceedings{Cle09:aardvark, + author = {Clement, Allen and Wong, Edmund and Alvisi, Lorenzo and Dahlin, Mike and Marchetti, Mirco}, + title = {Making Byzantine fault tolerant systems tolerate Byzantine faults}, + booktitle = {NSDI}, + year = {2009}, + pages = {153--168}, + numpages = {16} +} + +@inproceedings{Aiyer05:barB, + author = {Aiyer, Amitanand S. and Alvisi, Lorenzo and Clement, Allen and Dahlin, Mike and Martin, Jean-Philippe and Porth, Carl}, + title = {BAR fault tolerance for cooperative services}, + booktitle = {SOSP}, + year = {2005}, + pages = {45--58}, + numpages = {14} +} + +@inproceedings{Cach01:crypto, + author = {Cachin, Christian and Kursawe, Klaus and Petzold, Frank and Shoup, Victor}, + title = {Secure and Efficient Asynchronous Broadcast Protocols}, + booktitle = {CRYPTO}, + year = {2001}, + pages = {524--541}, + numpages = {18} +} + +@article{Moniz11:ritas, + author = {Moniz, Henrique and Neves, Nuno Ferreria and Correia, Miguel and Verissimo, Paulo}, + title = {RITAS: Services for Randomized Intrusion Tolerance}, + journal = {IEEE Trans. Dependable Secur. Comput.}, + volume = {8}, + number = {1}, + month = jan, + year = {2011}, + pages = {122--136}, + numpages = {15} +} + +@inproceedings{MHS11:jabc, + author = {Milosevic, Zarko and Hutle, Martin and Schiper, Andre}, + title = {On the Reduction of Atomic Broadcast to Consensus with Byzantine Faults}, + booktitle = {SRDS}, + year = {2011}, + pages = {235--244}, + numpages = {10} +} + +@incollection{DHSZ03, + author={Driscoll, Kevin and Hall, Brendan and Sivencrona, Håkan and Zumsteg, Phil}, + title={Byzantine Fault Tolerance, from Theory to Reality}, + year={2003}, + booktitle={Computer Safety, Reliability, and Security}, + volume={2788}, + pages={235--248} +} + +@inproceedings{RMES:dsn07, + author = {Olivier R{\"u}tti and + Sergio Mena and + Richard Ekwall and + Andr{\'e} Schiper}, + title = {On the Cost of Modularity in Atomic Broadcast}, + booktitle = {DSN}, + year = {2007}, + pages = {635-644} +} + +@article{Ben:jc92, + author = {Charles H. Bennett and + Fran\c{c}ois Bessette and + Gilles Brassard and + Louis Salvail and + John A. Smolin}, + title = {Experimental Quantum Cryptography}, + journal = {J. Cryptology}, + volume = {5}, + number = {1}, + year = {1992}, + pages = {3-28} +} + +@inproceedings{Aiyer:disc08, + author = {Aiyer, Amitanand S. and Alvisi, Lorenzo and Bazzi, Rida A. and Clement, Allen}, + title = {Matrix Signatures: From MACs to Digital Signatures in Distributed Systems}, + booktitle = {DISC}, + year = {2008}, + pages = {16--31}, + numpages = {16} +} + +@inproceedings{Biel13:dsn, + author = {Biely, Martin and Delgado, Pamela and Milosevic, Zarko and Schiper, Andr{\'e}}, + title = {Distal: A Framework for Implementing Fault-tolerant Distributed Algorithms}, + note = {To appear in DSN, 2013}, + year = 2013 +} + +@inproceedings{BS10:icdcn, + author = {Borran, Fatemeh and Schiper, Andr{\'e}}, + title = {A leader-free Byzantine consensus algorithm}, + booktitle = {ICDCN}, + year = {2010}, + pages = {67--78}, + numpages = {12} +} + +@article{Cor06:cj, + author = {Correia, Miguel and Neves, Nuno Ferreira and Ver\'{\i}ssimo, Paulo}, + title = {From Consensus to Atomic Broadcast: Time-Free Byzantine-Resistant Protocols without Signatures}, + journal = {Comput. J.}, + volume = {49}, + number = {1}, + year = {2006}, + pages = {82--96}, + numpages = {15} +} + +@inproceedings{RMS10:dsn, + author = {Olivier R{\"u}tti and + Zarko Milosevic and + Andr{\'e} Schiper}, + title = {Generic construction of consensus algorithms for benign + and Byzantine faults}, + booktitle = {DSN}, + year = {2010}, + pages = {343-352} +} + + + +@inproceedings{HKJR:usenix10, + author = {Hunt, Patrick and Konar, Mahadev and Junqueira, Flavio P. and Reed, Benjamin}, + title = {ZooKeeper: wait-free coordination for internet-scale systems}, + OPTbooktitle = {Proceedings of the 2010 USENIX conference on USENIX annual technical conference}, + booktitle = {USENIXATC}, + year = {2010}, + OPTlocation = {Boston, MA}, + pages = {11}, + numpages = {1}, + OPTurl = {http://dl.acm.org/citation.cfm?id=1855840.1855851}, + acmid = {1855851}, + OPTpublisher = {USENIX Association}, + OPTaddress = {Berkeley, CA, USA}, +} + +@inproceedings{Bur:osdi06, + author = {Burrows, Mike}, + title = {The Chubby lock service for loosely-coupled distributed systems}, + booktitle = {OSDI}, + year = {2006}, + pages = {335--350}, + numpages = {16}, +} + +@INPROCEEDINGS{Mao09:hotdep, + author = {Yanhua Mao and Flavio P. Junqueira and Keith Marzullo}, + title = {Towards low latency state machine replication for uncivil wide-area networks}, + booktitle = {HotDep}, + year = {2009} +} + +@inproceedings{Chun07:a2m, + author = {Chun, Byung-Gon and Maniatis, Petros and Shenker, Scott and Kubiatowicz, John}, + title = {Attested append-only memory: making adversaries stick to their word}, + booktitle = {SOSP}, + year = {2007}, + pages = {189--204}, + numpages = {16} +} + +@TECHREPORT{MBS:epfltr, + author = {Zarko Milosevic and Martin Biely and Andr\'e Schiper}, + title = {Bounded {D}elay in {B}yzantine {T}olerant {S}tate {M}achine {R}eplication}, + year = 2013, + month = april, + institution = {EPFL}, + number = {185962}, +} + +@book{BH09:datacenter, + author = {Barroso, Luiz Andre and Hoelzle, Urs}, + title = {The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines}, + year = {2009}, + isbn = {159829556X, 9781598295566}, + edition = {1st}, + publisher = {Morgan and Claypool Publishers}, +} + +@inproceedings{Kir11:csiirw, + author = {Kirsch, Jonathan and Goose, Stuart and Amir, Yair and Skare, Paul}, + title = {Toward survivable SCADA}, + booktitle = {CSIIRW}, + year = {2011}, + pages = {21:1--21:1}, + articleno = {21}, + numpages = {1} +} + +@inproceedings{Ongaro14:raft, + author = {Ongaro, Diego and Ousterhout, John}, + title = {In Search of an Understandable Consensus Algorithm}, + booktitle = {Proceedings of the 2014 USENIX Conference on USENIX Annual Technical Conference}, + series = {USENIX ATC'14}, + year = {2014}, + isbn = {978-1-931971-10-2}, + location = {Philadelphia, PA}, + pages = {305--320}, + numpages = {16}, + url = {http://dl.acm.org/citation.cfm?id=2643634.2643666}, + acmid = {2643666}, + publisher = {USENIX Association}, + address = {Berkeley, CA, USA}, +} + +@article{GLR17:red-belly-bc, + author = {Tyler Crain and + Vincent Gramoli and + Mikel Larrea and + Michel Raynal}, + title = {Leader/Randomization/Signature-free Byzantine Consensus for Consortium + Blockchains}, + journal = {CoRR}, + volume = {abs/1702.03068}, + year = {2017}, + url = {http://arxiv.org/abs/1702.03068}, + archivePrefix = {arXiv}, + eprint = {1702.03068}, + timestamp = {Wed, 07 Jun 2017 14:41:08 +0200}, + biburl = {http://dblp.org/rec/bib/journals/corr/CrainGLR17}, + bibsource = {dblp computer science bibliography, http://dblp.org} +} + + +@misc{Nak2012:bitcoin, + added-at = {2014-04-17T08:33:06.000+0200}, + author = {Nakamoto, Satoshi}, + biburl = {https://www.bibsonomy.org/bibtex/23db66df0fc9fa2b5033f096a901f1c36/ngnn}, + interhash = {423c2cdff70ba0cd0bca55ebb164d770}, + intrahash = {3db66df0fc9fa2b5033f096a901f1c36}, + keywords = {imported}, + timestamp = {2014-04-17T08:33:06.000+0200}, + title = {Bitcoin: A peer-to-peer electronic cash system}, + url = {http://www.bitcoin.org/bitcoin.pdf}, + year = 2009 +} + +@misc{But2014:ethereum, + author = {Vitalik Buterin}, + title = {Ethereum: A next-generation smart contract and decentralized application platform}, + year = {2014}, + howpublished = {\url{https://github.com/ethereum/wiki/wiki/White-Paper}}, + note = {Accessed: 2018-07-11}, + url = {https://github.com/ethereum/wiki/wiki/White-Paper}, +} + +@inproceedings{Dem1987:gossip, + author = {Demers, Alan and Greene, Dan and Hauser, Carl and Irish, Wes and Larson, John and Shenker, Scott and Sturgis, Howard and Swinehart, Dan and Terry, Doug}, + title = {Epidemic Algorithms for Replicated Database Maintenance}, + booktitle = {Proceedings of the Sixth Annual ACM Symposium on Principles of Distributed Computing}, + series = {PODC '87}, + year = {1987}, + isbn = {0-89791-239-X}, + location = {Vancouver, British Columbia, Canada}, + pages = {1--12}, + numpages = {12}, + url = {http://doi.acm.org/10.1145/41840.41841}, + doi = {10.1145/41840.41841}, + acmid = {41841}, + publisher = {ACM}, + address = {New York, NY, USA}, +} + +@article{Gue2018:sbft, + author = {Guy Golan{-}Gueta and + Ittai Abraham and + Shelly Grossman and + Dahlia Malkhi and + Benny Pinkas and + Michael K. Reiter and + Dragos{-}Adrian Seredinschi and + Orr Tamir and + Alin Tomescu}, + title = {{SBFT:} a Scalable Decentralized Trust Infrastructure for Blockchains}, + journal = {CoRR}, + volume = {abs/1804.01626}, + year = {2018}, + url = {http://arxiv.org/abs/1804.01626}, + archivePrefix = {arXiv}, + eprint = {1804.01626}, + timestamp = {Tue, 01 May 2018 19:46:29 +0200}, + biburl = {https://dblp.org/rec/bib/journals/corr/abs-1804-01626}, + bibsource = {dblp computer science bibliography, https://dblp.org} +} + +@inproceedings{BLS2001:crypto, + author = {Boneh, Dan and Lynn, Ben and Shacham, Hovav}, + title = {Short Signatures from the Weil Pairing}, + booktitle = {Proceedings of the 7th International Conference on the Theory and Application of Cryptology and Information Security: Advances in Cryptology}, + series = {ASIACRYPT '01}, + year = {2001}, + isbn = {3-540-42987-5}, + pages = {514--532}, + numpages = {19}, + url = {http://dl.acm.org/citation.cfm?id=647097.717005}, + acmid = {717005}, + publisher = {Springer-Verlag}, + address = {Berlin, Heidelberg}, +} + + diff --git a/cometbft/v0.38/spec/consensus/consensus-paper/paper.tex b/cometbft/v0.38/spec/consensus/consensus-paper/paper.tex new file mode 100644 index 00000000..22f8b405 --- /dev/null +++ b/cometbft/v0.38/spec/consensus/consensus-paper/paper.tex @@ -0,0 +1,153 @@ +%\documentclass[conference]{IEEEtran} +\documentclass[conference,onecolumn,draft,a4paper]{IEEEtran} +% Add the compsoc option for Computer Society conferences. +% +% If IEEEtran.cls has not been installed into the LaTeX system files, +% manually specify the path to it like: +% \documentclass[conference]{../sty/IEEEtran} + + + +% *** GRAPHICS RELATED PACKAGES *** +% +\ifCLASSINFOpdf +\else +\fi + +% correct bad hyphenation here +\hyphenation{op-tical net-works semi-conduc-tor} + +%\usepackage[caption=false,font=footnotesize]{subfig} +\usepackage{tikz} +\usetikzlibrary{decorations,shapes,backgrounds,calc} +\tikzstyle{msg}=[->,black,>=latex] +\tikzstyle{rubber}=[|<->|] +\tikzstyle{announce}=[draw=blue,fill=blue,shape=diamond,right,minimum + height=2mm,minimum width=1.6667mm,inner sep=0pt] +\tikzstyle{decide}=[draw=red,fill=red,shape=isosceles triangle,right,minimum + height=2mm,minimum width=1.6667mm,inner sep=0pt,shape border rotate=90] +\tikzstyle{cast}=[draw=green!50!black,fill=green!50!black,shape=circle,left,minimum + height=2mm,minimum width=1.6667mm,inner sep=0pt] + + +\usepackage{multirow} +\usepackage{graphicx} +\usepackage{epstopdf} +\usepackage{amssymb} +\usepackage{rounddiag} +\graphicspath{{../}} + +\usepackage{technote} +\usepackage{homodel} +\usepackage{enumerate} +%%\usepackage{ulem}\normalem + +% to center caption +\usepackage{caption} + +\newcommand{\textstretch}{1.4} +\newcommand{\algostretch}{1} +\newcommand{\eqnstretch}{0.5} + +\newconstruct{\FOREACH}{\textbf{for each}}{\textbf{do}}{\ENDFOREACH}{} + +%\newconstruct{\ON}{\textbf{on}}{\textbf{do}}{\ENDON}{\textbf{end on}} +\newcommand\With{\textbf{while}} +\newcommand\From{\textbf{from}} +\newcommand\Broadcast{\textbf{broadcast}} +\newcommand\PBroadcast{send} +\newcommand\UpCall{\textbf{UpCall}} +\newcommand\DownCall{\textbf{DownCall}} +\newcommand \Call{\textbf{Call}} +\newident{noop} +\newconstruct{\UPON}{\textbf{upon}}{\textbf{do}}{\ENDUPON}{} + + + +\newcommand{\abcast}{\mathsf{to\mbox{\sf-}broadcast}} +\newcommand{\adeliver}{\mathsf{to\mbox{\sf-}deliver}} + +\newcommand{\ABCAgreement}{\emph{TO-Agreement}} +\newcommand{\ABCIntegrity}{\emph{TO-Integrity}} +\newcommand{\ABCValidity}{\emph{TO-Validity}} +\newcommand{\ABCTotalOrder}{\emph{TO-Order}} +\newcommand{\ABCBoundedDelivery}{\emph{TO-Bounded Delivery}} + + +\newcommand{\tabc}{\mathit{atab\mbox{\sf-}cast}} +\newcommand{\anno}{\mathit{atab\mbox{\sf-}announce}} +\newcommand{\abort}{\mathit{atab\mbox{\sf-}abort}} +\newcommand{\tadel}{\mathit{atab\mbox{\sf-}deliver}} + +\newcommand{\ATABAgreement}{\emph{ATAB-Agreement}} +\newcommand{\ATABAbort}{\emph{ATAB-Abort}} +\newcommand{\ATABIntegrity}{\emph{ATAB-Integrity}} +\newcommand{\ATABValidity}{\emph{ATAB-Validity}} +\newcommand{\ATABAnnounce}{\emph{ATAB-Announcement}} +\newcommand{\ATABTermination}{\emph{ATAB-Termination}} +%\newcommand{\ATABFastAnnounce}{\emph{ATAB-Fast-Announcement}} + +%% Command for observations. +\newtheorem{observation}{Observation} + + +%% HO ALGORITHM DEFINITIONS +\newconstruct{\FUNCTION}{\textbf{Function}}{\textbf{:}}{\ENDFUNCTION}{} + +%% Uncomment the following four lines to remove remarks and visible traces of +%% modifications in the document +%%\renewcommand{\sout}[1]{\relaxx} +%%\renewcommand{\uline}[1]{#1} +%% \renewcommand{\uwave}[1]{#1} + \renewcommand{\note}[2][default]{\relax} + + +%% The following commands can be used to generate TR or Conference version of the paper +\newcommand{\tr}[1]{} +\renewcommand{\tr}[1]{#1} +\newcommand{\onlypaper}[1]{#1} +%\renewcommand{\onlypaper}[1]{} +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%\pagestyle{plain} +%\pagestyle{empty} + +%% IEEE tweaks +%\setlength{\IEEEilabelindent}{.5\parindent} +%\setlength{\IEEEiednormlabelsep}{.5\parindent} + +\begin{document} +% +% paper title +% can use linebreaks \\ within to get better formatting as desired +\title{The latest gossip on BFT consensus\vspace{-0.7\baselineskip}} + + + +\author{\IEEEauthorblockN{\large Ethan Buchman, Jae Kwon and Zarko Milosevic\\} + \IEEEauthorblockN{\large Tendermint}\\ + %\\\vspace{-0.5\baselineskip} + \IEEEauthorblockN{September 24, 2018} +} + +% make the title area +\maketitle +\vspace*{0.5em} + +\begin{abstract} +This paper presents Tendermint, a new protocol for ordering events in a distributed network under adversarial conditions. More commonly known as Byzantine Fault Tolerant (BFT) consensus or atomic broadcast, the problem has attracted significant attention in recent years due to the widespread success of blockchain-based digital currencies, such as Bitcoin and Ethereum, which successfully solved the problem in a public setting without a central authority. Tendermint modernizes classic academic work on the subject and simplifies the design of the BFT algorithm by relying on a peer-to-peer gossip protocol among nodes. +\end{abstract} + +%\noindent \textbf{Keywords:} Blockchain, Byzantine Fault Tolerance, State Machine %Replication + +\input{intro} +\input{definitions} +\input{consensus} +\input{proof} +\input{conclusion} + +\bibliographystyle{IEEEtran} +\bibliography{lit} + +%\appendix + +\end{document} diff --git a/cometbft/v0.38/spec/consensus/consensus-paper/proof.tex b/cometbft/v0.38/spec/consensus/consensus-paper/proof.tex new file mode 100644 index 00000000..1c84d9b1 --- /dev/null +++ b/cometbft/v0.38/spec/consensus/consensus-paper/proof.tex @@ -0,0 +1,280 @@ +\section{Proof of Tendermint consensus algorithm} \label{sec:proof} + +\begin{lemma} \label{lemma:majority-intersection} For all $f\geq 0$, any two +sets of processes with voting power at least equal to $2f+1$ have at least one +correct process in common. \end{lemma} + +\begin{proof} As the total voting power is equal to $n=3f+1$, we have $2(2f+1) + = n+f+1$. This means that the intersection of two sets with the voting + power equal to $2f+1$ contains at least $f+1$ voting power in common, \ie, + at least one correct process (as the total voting power of faulty processes + is $f$). The result follows directly from this. \end{proof} + +\begin{lemma} \label{lemma:locked-decision_value-prevote-v} If $f+1$ correct +processes lock value $v$ in round $r_0$ ($lockedValue = v$ and $lockedRound = +r_0$), then in all rounds $r > r_0$, they send $\Prevote$ for $id(v)$ or +$\nil$. \end{lemma} + +\begin{proof} We prove the result by induction on $r$. + +\emph{Base step $r = r_0 + 1:$} Let's denote with $C$ the set of correct +processes with voting power equal to $f+1$. By the rules at +line~\ref{line:tab:recvProposal} and line~\ref{line:tab:acceptProposal}, the +processes from the set $C$ can't accept $\Proposal$ for any value different +from $v$ in round $r$, and therefore can't send a $\li{\Prevote,height_p, +r,id(v')}$ message, if $v' \neq v$. Therefore, the Lemma holds for the base +step. + +\emph{Induction step from $r_1$ to $r_1+1$:} We assume that no process from the +set $C$ has sent $\Prevote$ for values different than $id(v)$ or $\nil$ until +round $r_1 + 1$. We now prove that the Lemma also holds for round $r_1 + 1$. As +processes from the set $C$ send $\Prevote$ for $id(v)$ or $\nil$ in rounds $r_0 +\le r \le r_1$, by Lemma~\ref{lemma:majority-intersection} there is no value +$v' \neq v$ for which it is possible to receive $2f+1$ $\Prevote$ messages in +those rounds (i). Therefore, we have for all processes from the set $C$, +$lockedValue = v$ and $lockedRound \ge r_0$. Let's assume by a contradiction +that a process $q$ from the set $C$ sends $\Prevote$ in round $r_1 + 1$ for +value $id(v')$, where $v' \neq v$. This is possible only by +line~\ref{line:tab:prevote-higher-proposal}. Note that this implies that $q$ +received $2f+1$ $\li{\Prevote,h_q, r,id(v')}$ messages, where $r > r_0$ and $r +< r_1 +1$ (see line~\ref{line:tab:cond-prevote-higher-proposal}). A +contradiction with (i) and Lemma~\ref{lemma:majority-intersection}. +\end{proof} + +\begin{lemma} \label{lemma:agreement} Algorithm~\ref{alg:tendermint} satisfies +Agreement. \end{lemma} + +\begin{proof} Let round $r_0$ be the first round of height $h$ such that some + correct process $p$ decides $v$. We now prove that if some correct process + $q$ decides $v'$ in some round $r \ge r_0$, then $v = v'$. + +In case $r = r_0$, $q$ has received at least $2f+1$ +$\li{\Precommit,h_p,r_0,id(v')}$ messages at line~\ref{line:tab:onDecideRule}, +while $p$ has received at least $2f+1$ $\li{\Precommit,h_p,r_0,id(v)}$ +messages. By Lemma~\ref{lemma:majority-intersection} two sets of messages of +voting power $2f+1$ intersect in at least one correct process. As a correct +process sends a single $\Precommit$ message in a round, then $v=v'$. + +We prove the case $r > r_0$ by contradiction. By the +rule~\ref{line:tab:onDecideRule}, $p$ has received at least $2f+1$ voting-power +equivalent of $\li{\Precommit,h_p,r_0,id(v)}$ messages, i.e., at least $f+1$ +voting-power equivalent correct processes have locked value $v$ in round $r_0$ and have +sent those messages (i). Let denote this set of messages with $C$. On the +other side, $q$ has received at least $2f+1$ voting power equivalent of +$\li{\Precommit,h_q, r,id(v')}$ messages. As the voting power of all faulty +processes is at most $f$, some correct process $c$ has sent one of those +messages. By the rule at line~\ref{line:tab:recvPrevote}, $c$ has locked value +$v'$ in round $r$ before sending $\li{\Precommit,h_q, r,id(v')}$. Therefore $c$ +has received $2f+1$ $\Prevote$ messages for $id(v')$ in round $r > r_0$ (see +line~\ref{line:tab:recvPrevote}). By Lemma~\ref{lemma:majority-intersection}, a +process from the set $C$ has sent $\Prevote$ message for $id(v')$ in round $r$. +A contradiction with (i) and Lemma~\ref{lemma:locked-decision_value-prevote-v}. +\end{proof} + +\begin{lemma} \label{lemma:agreement} Algorithm~\ref{alg:tendermint} satisfies +Validity. \end{lemma} + +\begin{proof} Trivially follows from the rule at line +\ref{line:tab:validDecisionValue} which ensures that only valid values can be +decided. \end{proof} + +\begin{lemma} \label{lemma:round-synchronisation} If we assume that: +\begin{enumerate} + \item a correct process $p$ is the first correct process to + enter a round $r>0$ at time $t > GST$ (for every correct process + $c$, $round_c \le r$ at time $t$) + \item the proposer of round $r$ is + a correct process $q$ + \item for every correct process $c$, + $lockedRound_c \le validRound_q$ at time $t$ + \item $\timeoutPropose(r) + > 2\Delta + \timeoutPrecommit(r-1)$, $\timeoutPrevote(r) > 2\Delta$ and + $\timeoutPrecommit(r) > 2\Delta$, +\end{enumerate} +then all correct processes decide in round $r$ before $t + 4\Delta + + \timeoutPrecommit(r-1)$. +\end{lemma} + +\begin{proof} As $p$ is the first correct process to enter round $r$, it + executed the line~\ref{line:tab:nextRound} after $\timeoutPrecommit(r-1)$ + expired. Therefore, $p$ received $2f+1$ $\Precommit$ messages in the round + $r-1$ before time $t$. By the \emph{Gossip communication} property, all + correct processes will receive those messages the latest at time $t + + \Delta$. Correct processes that are in rounds $< r-1$ at time $t$ will + enter round $r-1$ (see the rule at line~\ref{line:tab:nextRound2}) and + trigger $\timeoutPrecommit(r-1)$ (see rule~\ref{line:tab:startTimeoutPrecommit}) + by time $t+\Delta$. Therefore, all correct processes will start round $r$ + by time $t+\Delta+\timeoutPrecommit(r-1)$ (i). + +In the worst case, the process $q$ is the last correct process to enter round +$r$, so $q$ starts round $r$ and sends $\Proposal$ message for some value $v$ +at time $t + \Delta + \timeoutPrecommit(r-1)$. Therefore, all correct processes +receive the $\Proposal$ message from $q$ the latest by time $t + 2\Delta + +\timeoutPrecommit(r-1)$. Therefore, if $\timeoutPropose(r) > 2\Delta + +\timeoutPrecommit(r-1)$, all correct processes will receive $\Proposal$ message +before $\timeoutPropose(r)$ expires. + +By (3) and the rules at line~\ref{line:tab:recvProposal} and +\ref{line:tab:acceptProposal}, all correct processes will accept the +$\Proposal$ message for value $v$ and will send a $\Prevote$ message for +$id(v)$ by time $t + 2\Delta + \timeoutPrecommit(r-1)$. Note that by the +\emph{Gossip communication} property, the $\Prevote$ messages needed to trigger +the rule at line~\ref{line:tab:acceptProposal} are received before time $t + +\Delta$. + +By time $t + 3\Delta + \timeoutPrecommit(r-1)$, all correct processes will receive +$\Proposal$ for $v$ and $2f+1$ corresponding $\Prevote$ messages for $id(v)$. +By the rule at line~\ref{line:tab:recvPrevote}, all correct processes will send +a $\Precommit$ message (see line~\ref{line:tab:precommit-v}) for $id(v)$ by +time $t + 3\Delta + \timeoutPrecommit(r-1)$. Therefore, by time $t + 4\Delta + +\timeoutPrecommit(r-1)$, all correct processes will have received the $\Proposal$ +for $v$ and $2f+1$ $\Precommit$ messages for $id(v)$, so they decide at +line~\ref{line:tab:decide} on $v$. + +This scenario holds if every correct process $q$ sends a $\Precommit$ message +before $\timeoutPrevote(r)$ expires, and if $\timeoutPrecommit(r)$ does not expire +before $t + 4\Delta + \timeoutPrecommit(r-1)$. Let's assume that a correct process +$c_1$ is the first correct process to trigger $\timeoutPrevote(r)$ (see the rule +at line~\ref{line:tab:recvAny2/3Prevote}) at time $t_1 > t$. This implies that +before time $t_1$, $c_1$ received a $\Proposal$ ($step_{c_1}$ must be +$\prevote$ by the rule at line~\ref{line:tab:recvAny2/3Prevote}) and a set of +$2f+1$ $\Prevote$ messages. By time $t_1 + \Delta$, all correct processes will +receive those messages. Note that even if some correct process was in the +smaller round before time $t_1$, at time $t_1 + \Delta$ it will start round $r$ +after receiving those messages (see the rule at +line~\ref{line:tab:skipRounds}). Therefore, all correct processes will send +their $\Prevote$ message for $id(v)$ by time $t_1 + \Delta$, and all correct +processes will receive those messages the by time $t_1 + 2\Delta$. Therefore, +as $\timeoutPrevote(r) > 2\Delta$, this ensures that all correct processes receive +$\Prevote$ messages from all correct processes before their respective local +$\timeoutPrevote(r)$ expire. + +On the other hand, $\timeoutPrecommit(r)$ is triggered in a correct process $c_2$ +after it receives any set of $2f+1$ $\Precommit$ messages for the first time. +Let's denote with $t_2 > t$ the earliest point in time $\timeoutPrecommit(r)$ is +triggered in some correct process $c_2$. This implies that $c_2$ has received +at least $f+1$ $\Precommit$ messages for $id(v)$ from correct processes, i.e., +those processes have received $\Proposal$ for $v$ and $2f+1$ $\Prevote$ +messages for $id(v)$ before time $t_2$. By the \emph{Gossip communication} +property, all correct processes will receive those messages by time $t_2 + +\Delta$, and will send $\Precommit$ messages for $id(v)$. Note that even if +some correct processes were at time $t_2$ in a round smaller than $r$, by the +rule at line~\ref{line:tab:skipRounds} they will enter round $r$ by time $t_2 + +\Delta$. Therefore, by time $t_2 + 2\Delta$, all correct processes will +receive $\Proposal$ for $v$ and $2f+1$ $\Precommit$ messages for $id(v)$. So if +$\timeoutPrecommit(r) > 2\Delta$, all correct processes will decide before the +timeout expires. \end{proof} + + +\begin{lemma} \label{lemma:validValue} If a correct process $p$ locks a value + $v$ at time $t_0 > GST$ in some round $r$ ($lockedValue = v$ and + $lockedRound = r$) and $\timeoutPrecommit(r) > 2\Delta$, then all correct + processes set $validValue$ to $v$ and $validRound$ to $r$ before starting + round $r+1$. \end{lemma} + +\begin{proof} In order to prove this Lemma, we need to prove that if the + process $p$ locks a value $v$ at time $t_0$, then no correct process will + leave round $r$ before time $t_0 + \Delta$ (unless it has already set + $validValue$ to $v$ and $validRound$ to $r$). It is sufficient to prove + this, since by the \emph{Gossip communication} property the messages that + $p$ received at time $t_0$ and that triggered rule at + line~\ref{line:tab:recvPrevote} will be received by time $t_0 + \Delta$ by + all correct processes, so all correct processes that are still in round $r$ + will set $validValue$ to $v$ and $validRound$ to $r$ (by the rule at + line~\ref{line:tab:recvPrevote}). To prove this, we need to compute the + earliest point in time a correct process could leave round $r$ without + updating $validValue$ to $v$ and $validRound$ to $r$ (we denote this time + with $t_1$). The Lemma is correct if $t_0 + \Delta < t_1$. + +If the process $p$ locks a value $v$ at time $t_0$, this implies that $p$ +received the valid $\Proposal$ message for $v$ and $2f+1$ +$\li{\Prevote,h,r,id(v)}$ at time $t_0$. At least $f+1$ of those messages are +sent by correct processes. Let's denote this set of correct processes as $C$. By +Lemma~\ref{lemma:majority-intersection} any set of $2f+1$ $\Prevote$ messages +in round $r$ contains at least a single message from the set $C$. + +Let's denote as time $t$ the earliest point in time a correct process, $c_1$, triggered +$\timeoutPrevote(r)$. This implies that $c_1$ received $2f+1$ $\Prevote$ messages +(see the rule at line \ref{line:tab:recvAny2/3Prevote}), where at least one of +those messages was sent by a process $c_2$ from the set $C$. Therefore, process +$c_2$ had received $\Proposal$ message before time $t$. By the \emph{Gossip +communication} property, all correct processes will receive $\Proposal$ and +$2f+1$ $\Prevote$ messages for round $r$ by time $t+\Delta$. The latest point +in time $p$ will trigger $\timeoutPrevote(r)$ is $t+\Delta$\footnote{Note that +even if $p$ was in smaller round at time $t$ it will start round $r$ by time +$t+\Delta$.}. So the latest point in time $p$ can lock the value $v$ in +round $r$ is $t_0 = t+\Delta+\timeoutPrevote(r)$ (as at this point +$\timeoutPrevote(r)$ expires, so a process sends $\Precommit$ $\nil$ and updates +$step$ to $\precommit$, see line \ref{line:tab:onTimeoutPrevote}). + +Note that according to the Algorithm \ref{alg:tendermint}, a correct process +can not send a $\Precommit$ message before receiving $2f+1$ $\Prevote$ +messages. Therefore, no correct process can send a $\Precommit$ message in +round $r$ before time $t$. If a correct process sends a $\Precommit$ message +for $\nil$, it implies that it has waited for the full duration of +$\timeoutPrevote(r)$ (see line +\ref{line:tab:precommit-nil-onTimeout})\footnote{The other case in which a +correct process $\Precommit$ for $\nil$ is after receiving $2f+1$ $Prevote$ for +$\nil$ messages, see the line \ref{line:tab:precommit-v-1}. By +Lemma~\ref{lemma:majority-intersection}, this is not possible in round $r$.}. +Therefore, no correct process can send $\Precommit$ for $\nil$ before time $t + +\timeoutPrevote(r)$ (*). + +A correct process $q$ that enters round $r+1$ must wait (i) $\timeoutPrecommit(r)$ +(see line \ref{line:tab:nextRound}) or (ii) receiving $f+1$ messages from the +round $r+1$ (see the line \ref{line:tab:skipRounds}). In the former case, $q$ +receives $2f+1$ $\Precommit$ messages before starting $\timeoutPrecommit(r)$. If +at least a single $\Precommit$ message from a correct process (at least $f+1$ +voting power equivalent of those messages is sent by correct processes) is for +$\nil$, then $q$ cannot start round $r+1$ before time $t_1 = t + +\timeoutPrevote(r) + \timeoutPrecommit(r)$ (see (*)). Therefore in this case we have: +$t_0 + \Delta < t_1$, i.e., $t+2\Delta+\timeoutPrevote(r) < t + \timeoutPrevote(r) + +\timeoutPrecommit(r)$, and this is true whenever $\timeoutPrecommit(r) > 2\Delta$, so +Lemma holds in this case. + +If in the set of $2f+1$ $\Precommit$ messages $q$ receives, there is at least a +single $\Precommit$ for $id(v)$ message from a correct process $c$, then $q$ +can start the round $r+1$ the earliest at time $t_1 = t+\timeoutPrecommit(r)$. In +this case, by the \emph{Gossip communication} property, all correct processes +will receive $\Proposal$ and $2f+1$ $\Prevote$ messages (that $c$ received +before time $t$) the latest at time $t+\Delta$. Therefore, $q$ will set +$validValue$ to $v$ and $validRound$ to $r$ the latest at time $t+\Delta$. As +$t+\Delta < t+\timeoutPrecommit(r)$, whenever $\timeoutPrecommit(r) > \Delta$, the +Lemma holds also in this case. + +In case (ii), $q$ received at least a single message from a correct process $c$ +from the round $r+1$. The earliest point in time $c$ could have started round +$r+1$ is $t+\timeoutPrecommit(r)$ in case it received a $\Precommit$ message for +$v$ from some correct process in the set of $2f+1$ $\Precommit$ messages it +received. The same reasoning as above holds also in this case, so $q$ set +$validValue$ to $v$ and $validRound$ to $r$ the latest by time $t+\Delta$. As +$t+\Delta < t+\timeoutPrecommit(r)$, whenever $\timeoutPrecommit(r) > \Delta$, the +Lemma holds also in this case. \end{proof} + +\begin{lemma} \label{lemma:agreement} Algorithm~\ref{alg:tendermint} satisfies +Termination. \end{lemma} + +\begin{proof} Lemma~\ref{lemma:round-synchronisation} defines a scenario in + which all correct processes decide. We now prove that within a bounded + duration after GST such a scenario will unfold. Let's assume that at time + $GST$ the highest round started by a correct process is $r_0$, and that + there exists a correct process $p$ such that the following holds: for every + correct process $c$, $lockedRound_c \le validRound_p$. Furthermore, we + assume that $p$ will be the proposer in some round $r_1 > r$ (this is + ensured by the $\coord$ function). + +We have two cases to consider. In the first case, for all rounds $r \ge r_0$ +and $r < r_1$, no correct process locks a value (set $lockedRound$ to $r$). So +in round $r_1$ we have the scenario from the +Lemma~\ref{lemma:round-synchronisation}, so all correct processes decides in +round $r_1$. + +In the second case, a correct process locks a value $v$ in round $r_2$, where +$r_2 \ge r_0$ and $r_2 < r_1$. Let's assume that $r_2$ is the highest round +before $r_1$ in which some correct process $q$ locks a value. By Lemma +\ref{lemma:validValue} at the end of round $r_2$ the following holds for all +correct processes $c$: $validValue_c = lockedValue_q$ and $validRound_c = r_2$. +Then in round $r_1$, the conditions for the +Lemma~\ref{lemma:round-synchronisation} holds, so all correct processes decide. +\end{proof} + diff --git a/cometbft/v0.38/spec/consensus/consensus-paper/rounddiag.sty b/cometbft/v0.38/spec/consensus/consensus-paper/rounddiag.sty new file mode 100644 index 00000000..a6ca5d88 --- /dev/null +++ b/cometbft/v0.38/spec/consensus/consensus-paper/rounddiag.sty @@ -0,0 +1,62 @@ +% ROUNDDIAG STYLE +% for LaTeX version 2e +% by -- 2008 Martin Hutle +% +% This style file is free software; you can redistribute it and/or +% modify it under the terms of the GNU Lesser General Public +% License as published by the Free Software Foundation; either +% version 2 of the License, or (at your option) any later version. +% +% This style file is distributed in the hope that it will be useful, +% but WITHOUT ANY WARRANTY; without even the implied warranty of +% MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +% Lesser General Public License for more details. +% +% You should have received a copy of the GNU Lesser General Public +% License along with this style file; if not, write to the +% Free Software Foundation, Inc., 59 Temple Place - Suite 330, +% Boston, MA 02111-1307, USA. +% +\NeedsTeXFormat{LaTeX2e} +\ProvidesPackage{rounddiag} +\typeout{Document Style `rounddiag' - provides simple round diagrams} +% +\RequirePackage{ifthen} +\RequirePackage{calc} +\RequirePackage{tikz} + +\def\rdstretch{3} + +\tikzstyle{msg}=[->,thick,>=latex] +\tikzstyle{rndline}=[dotted] +\tikzstyle{procline}=[dotted] + +\newenvironment{rounddiag}[2]{ +\begin{center} +\begin{tikzpicture} +\foreach \i in {1,...,#1}{ + \draw[procline] (0,#1-\i) node[xshift=-1em]{$p_{\i}$} -- (#2*\rdstretch+1,#1-\i); +} +\foreach \i in {0,...,#2}{ + \draw[rndline] (\i*\rdstretch+0.5,0) -- (\i*\rdstretch+0.5,#1-1); +} +\newcommand{\rdat}[2]{ + (##2*\rdstretch+0.5,#1-##1) +}% +\newcommand{\round}[2]{% + \def\rdround{##1} + \ifthenelse{\equal{##2}{}}{}{ + \node[yshift=-1em] at ({##1*\rdstretch+0.5-0.5*\rdstretch},0) {##2}; + } +}% +\newcommand{\rdmessage}[3]{\draw[msg] + (\rdround*\rdstretch-\rdstretch+0.5,#1-##1) -- node[yshift=1.2ex]{##3} + (\rdround*\rdstretch+0.5,#1-##2);}% +\newcommand{\rdalltoall}{% + \foreach \i in {1,...,#1}{ + \foreach \j in {1,...,#1}{ + { \rdmessage{\i}{\j}{}}}}}% +}{% +\end{tikzpicture} +\end{center} +} diff --git a/cometbft/v0.38/spec/consensus/consensus-paper/technote.sty b/cometbft/v0.38/spec/consensus/consensus-paper/technote.sty new file mode 100644 index 00000000..5353f13c --- /dev/null +++ b/cometbft/v0.38/spec/consensus/consensus-paper/technote.sty @@ -0,0 +1,118 @@ +\NeedsTeXFormat{LaTeX2e} +\ProvidesPackage{technote}[2007/11/09] +\typeout{Template for quick notes with some useful definitions} + +\RequirePackage{ifthen} +\RequirePackage{calc} +\RequirePackage{amsmath,amssymb,amsthm} +\RequirePackage{epsfig} +\RequirePackage{algorithm} +\RequirePackage[noend]{algorithmicplus} + +\newboolean{technote@noedit} +\setboolean{technote@noedit}{false} +\DeclareOption{noedit}{\setboolean{technote@noedit}{true}} + +\newcounter{technote@lang} +\setcounter{technote@lang}{0} +\DeclareOption{german}{\setcounter{technote@lang}{1}} +\DeclareOption{french}{\setcounter{technote@lang}{2}} + +\DeclareOption{fullpage}{ +\oddsidemargin -10mm % Margin on odd side pages (default=0mm) +\evensidemargin -10mm % Margin on even side pages (default=0mm) +\topmargin -10mm % Top margin space (default=16mm) +\headheight \baselineskip % Height of headers (default=0mm) +\headsep \baselineskip % Separation spc btw header and text (d=0mm) +\footskip 30pt % Separation spc btw text and footer (d=30pt) +\textheight 230mm % Total text height (default=200mm) +\textwidth 180mm % Total text width (default=160mm) +} + +\renewcommand{\algorithmiccomment}[1]{\hfill/* #1 */} +\renewcommand{\algorithmiclnosize}{\scriptsize} + +\newboolean{technote@truenumbers} +\setboolean{technote@truenumbers}{false} +\DeclareOption{truenumbers}{\setboolean{technote@truenumbers}{true}} + +\ProcessOptions + +\newcommand{\N}{\ifthenelse{\boolean{technote@truenumbers}}% + {\mbox{\rm I\hspace{-.5em}N}}% + {\mathbb{N}}} + +\newcommand{\R}{\ifthenelse{\boolean{technote@truenumbers}}% + {\mbox{\rm I\hspace{-.2em}R}}% + {\mathbb{R}}} + +\newcommand{\Z}{\mathbb{Z}} + +\newcommand{\set}[1]{\left\{#1\right\}} +\newcommand{\mathsc}[1]{\mbox{\sc #1}} +\newcommand{\li}[1]{\langle#1\rangle} +\newcommand{\st}{\;s.t.\;} +\newcommand{\Real}{\R} +\newcommand{\Natural}{\N} +\newcommand{\Integer}{\Z} + +% edit commands +\newcommand{\newedit}[2]{ + \newcommand{#1}[2][default]{% + \ifthenelse{\boolean{technote@noedit}}{}{ + \par\vspace{2mm} + \noindent + \begin{tabular}{|l|}\hline + \parbox{\linewidth-\tabcolsep*2}{{\bf #2:}\hfill\ifthenelse{\equal{##1}{default}}{}{##1}}\\\hline + \parbox{\linewidth-\tabcolsep*2}{\rule{0pt}{5mm}##2\rule[-2mm]{0pt}{2mm}}\\\hline + \end{tabular} + \par\vspace{2mm} + } + } +} + +\newedit{\note}{Note} +\newedit{\comment}{Comment} +\newedit{\question}{Question} +\newedit{\content}{Content} +\newedit{\problem}{Problem} + +\newcommand{\mnote}[1]{\marginpar{\scriptsize\it + \begin{minipage}[t]{0.8 in} + \raggedright #1 + \end{minipage}}} + +\newcommand{\Insert}[1]{\underline{#1}\marginpar{$|$}} + +\newcommand{\Delete}[1]{\marginpar{$|$} +} + +% lemma, theorem, etc. +\newtheorem{lemma}{Lemma} +\newtheorem{proposition}{Proposition} +\newtheorem{theorem}{Theorem} +\newtheorem{corollary}{Corollary} +\newtheorem{assumption}{Assumption} +\newtheorem{definition}{Definition} + +\gdef\op|{\,|\;} +\gdef\op:{\,:\;} +\newcommand{\assign}{\leftarrow} +\newcommand{\inc}[1]{#1 \assign #1 + 1} +\newcommand{\isdef}{:=} + +\newcommand{\ident}[1]{\mathit{#1}} +\def\newident#1{\expandafter\def\csname #1\endcsname{\ident{#1}}} + +\newcommand{\eg}{{\it e.g.}} +\newcommand{\ie}{{\it i.e.}} +\newcommand{\apriori}{{\it apriori}} +\newcommand{\etal}{{\it et al.}} + +\newcommand\ps@technote{% + \renewcommand\@oddhead{\theheader}% + \let\@evenhead\@oddhead + \renewcommand\@evenfoot + {\hfil\normalfont\textrm{\thepage}\hfil}% + \let\@oddfoot\@evenfoot +} diff --git a/cometbft/v0.38/spec/consensus/light-client/accountability.md b/cometbft/v0.38/spec/consensus/light-client/accountability.md new file mode 100644 index 00000000..3907e8d4 --- /dev/null +++ b/cometbft/v0.38/spec/consensus/light-client/accountability.md @@ -0,0 +1,3 @@ +# Fork accountability + +Deprecated, please see [light-client/accountability](https://github.com/cometbft/cometbft/blob/v0.38.x/spec/light-client/accountability). diff --git a/cometbft/v0.38/spec/consensus/light-client/assets/light-node-image.png b/cometbft/v0.38/spec/consensus/light-client/assets/light-node-image.png new file mode 100644 index 00000000..f0b93c6e Binary files /dev/null and b/cometbft/v0.38/spec/consensus/light-client/assets/light-node-image.png differ diff --git a/cometbft/v0.38/spec/consensus/light-client/detection.md b/cometbft/v0.38/spec/consensus/light-client/detection.md new file mode 100644 index 00000000..9e70726c --- /dev/null +++ b/cometbft/v0.38/spec/consensus/light-client/detection.md @@ -0,0 +1,3 @@ +# Detection + +Deprecated, please see [light-client/detection](https://github.com/cometbft/cometbft/blob/v0.38.x/spec/light-client/detection). diff --git a/cometbft/v0.38/spec/consensus/light-client/verification.md b/cometbft/v0.38/spec/consensus/light-client/verification.md new file mode 100644 index 00000000..d0e2bf1e --- /dev/null +++ b/cometbft/v0.38/spec/consensus/light-client/verification.md @@ -0,0 +1,3 @@ +# Core Verification + +Deprecated, please see [light-client/verification](https://github.com/cometbft/cometbft/blob/v0.38.x/spec/light-client/verification). diff --git a/cometbft/v0.38/spec/consensus/proposer-based-timestamp/README.md b/cometbft/v0.38/spec/consensus/proposer-based-timestamp/README.md new file mode 100644 index 00000000..2972d876 --- /dev/null +++ b/cometbft/v0.38/spec/consensus/proposer-based-timestamp/README.md @@ -0,0 +1,20 @@ +# Proposer-Based Timestamps + +This section describes a version of the Tendermint consensus algorithm, adopted in CometBFT, +which uses proposer-based timestamps. + +## Contents + +- [Proposer-Based Time][main] (entry point) +- [Part I - System Model and Properties][sysmodel] +- [Part II - Protocol Specification][algorithm] +- [TLA+ Specification][proposertla] + + +[algorithm]: ./pbts-algorithm_001_draft.md + +[sysmodel]: ./pbts-sysmodel_001_draft.md + +[main]: ./pbts_001_draft.md + +[proposertla]: ./tla/TendermintPBT_001_draft.tla diff --git a/cometbft/v0.38/spec/consensus/proposer-based-timestamp/pbts-algorithm_001_draft.md b/cometbft/v0.38/spec/consensus/proposer-based-timestamp/pbts-algorithm_001_draft.md new file mode 100644 index 00000000..ee8ca693 --- /dev/null +++ b/cometbft/v0.38/spec/consensus/proposer-based-timestamp/pbts-algorithm_001_draft.md @@ -0,0 +1,160 @@ +# Proposer-Based Time - Part II + +## Updated Consensus Algorithm + +### Outline + +The algorithm in the [arXiv paper][arXiv] evaluates rules of the received messages without making explicit how these messages are received. In our solution, we will make some message filtering explicit. We will assume that there are message reception steps (where messages are received and possibly stored locally for later evaluation of rules) and processing steps (the latter roughly as described in a way similar to the pseudo code of the arXiv paper). + +In contrast to the original algorithm the field `proposal` in the `PROPOSE` message is a pair `(v, time)`, of the proposed consensus value `v` and the proposed time `time`. + +#### **[PBTS-RECEPTION-STEP.0]** + +In the reception step at process `p` at local time `now_p`, upon receiving a message `m`: + +- if the message `m` is of type `PROPOSE` and satisfies `now_p - PRECISION < m.time < now_p + PRECISION + MSGDELAY`, then mark the message as `timely` + +> if `m` does not satisfy the constraint consider it `untimely` + + +#### **[PBTS-PROCESSING-STEP.0]** + +In the processing step, based on the messages stored, the rules of the algorithms are +executed. Note that the processing step only operates on messages +for the current height. The consensus algorithm rules are defined by the following updates to arXiv paper. + +#### New `StartRound` + +There are two additions + +- in case the proposer's local time is smaller than the time of the previous block, the proposer waits until this is not the case anymore (to ensure the block time is monotonically increasing) +- the proposer sends its time `now_p` as part of its proposal + +We update the timeout for the `PROPOSE` step according to the following reasoning: + +- If a correct proposer needs to wait to make sure its proposed time is larger than the `blockTime` of the previous block, then it sends by realtime `blockTime + ACCURACY` (By this time, its local clock must exceed `blockTime`) +- the receiver will receive a `PROPOSE` message by `blockTime + ACCURACY + MSGDELAY` +- the receiver's local clock will be `<= blockTime + 2 * ACCURACY + MSGDELAY` +- thus when the receiver `p` enters this round it can set its timeout to a value `waitingTime => blockTime + 2 * ACCURACY + MSGDELAY - now_p` + +So we should set the timeout to `max(timeoutPropose(round_p), waitingTime)`. + +> If, in the future, a block delay parameter `BLOCKDELAY` is introduced, this means +that the proposer should wait for `now_p > blockTime + BLOCKDELAY` before sending a `PROPOSE` message. +Also, `BLOCKDELAY` needs to be added to `waitingTime`. + +#### **[PBTS-ALG-STARTROUND.0]** + +```go +function StartRound(round) { + blockTime ← block time of block h_p - 1 + waitingTime ← blockTime + 2 * ACCURACY + MSGDELAY - now_p + round_p ← round + step_p ← propose + if proposer(h_p, round_p) = p { + wait until now_p > blockTime // new wait condition + if validValue_p != nil { + proposal ← (validValue_p, now_p) // added "now_p" + } + else { + proposal ← (getValue(), now_p) // added "now_p" + } + broadcast ⟨PROPOSAL, h_p, round_p, proposal, validRound_p⟩ + } + else { + schedule OnTimeoutPropose(h_p,round_p) to be executed after max(timeoutPropose(round_p), waitingTime) + } +} +``` + +#### New Rule Replacing Lines 22 - 27 + +- a validator prevotes for the consensus value `v` **and** the time `t` +- the code changes as the `PROPOSAL` message carries time (while `lockedValue` does not) + +#### **[PBTS-ALG-UPON-PROP.0]** + +```go +upon timely(⟨PROPOSAL, h_p, round_p, (v,t), −1⟩) from proposer(h_p, round_p) while step_p = propose do { + if valid(v) ∧ (lockedRound_p = −1 ∨ lockedValue_p = v) { + broadcast ⟨PREVOTE, h_p, round_p, id(v,t)⟩ + } + else { + broadcast ⟨PREVOTE, h_p, round_p, nil⟩ + } + step_p ← prevote +} +``` + +#### New Rule Replacing Lines 28 - 33 + +In case consensus is not reached in round 1, in `StartRound` the proposer of future rounds may propose the same value but with a different time. +Thus, the time `tprop` in the `PROPOSAL` message need not match the time `tvote` in the (old) `PREVOTE` messages. +A validator may send `PREVOTE` for the current round as long as the value `v` matches. +This gives the following rule: + +#### **[PBTS-ALG-OLD-PREVOTE.0]** + +```go +upon timely(⟨PROPOSAL, h_p, round_p, (v, tprop), vr⟩) from proposer(h_p, round_p) AND 2f + 1 ⟨PREVOTE, h_p, vr, id((v, tvote)⟩ +while step_p = propose ∧ (vr ≥ 0 ∧ vr < round_p) do { + if valid(v) ∧ (lockedRound_p ≤ vr ∨ lockedValue_p = v) { + broadcast ⟨PREVOTE, h_p, roundp, id(v, tprop)⟩ + } + else { + broadcast ⟨PREVOTE, hp, roundp, nil⟩ + } + step_p ← prevote +} +``` + +#### New Rule Replacing Lines 36 - 43 + +- As above, in the following `(v,t)` is part of the message rather than `v` +- the stored values (i.e., `lockedValue`, `validValue`) do not contain the time + +#### **[PBTS-ALG-NEW-PREVOTE.0]** + +```go +upon timely(⟨PROPOSAL, h_p, round_p, (v,t), ∗⟩) from proposer(h_p, round_p) AND 2f + 1 ⟨PREVOTE, h_p, round_p, id(v,t)⟩ while valid(v) ∧ step_p ≥ prevote for the first time do { + if step_p = prevote { + lockedValue_p ← v + lockedRound_p ← round_p + broadcast ⟨PRECOMMIT, h_p, round_p, id(v,t))⟩ + step_p ← precommit + } + validValue_p ← v + validRound_p ← round_p +} +``` + +#### New Rule Replacing Lines 49 - 54 + +- we decide on `v` as well as on the time from the proposal message +- here we do not care whether the proposal was received timely. + +> In particular we need to take care of the case where the proposer is untimely to one correct validator only. We need to ensure that this validator decides if all decide. + +#### **[PBTS-ALG-DECIDE.0]** + +```go +upon ⟨PROPOSAL, h_p, r, (v,t), ∗⟩ from proposer(h_p, r) AND 2f + 1 ⟨PRECOMMIT, h_p, r, id(v,t)⟩ while decisionp[h_p] = nil do { + if valid(v) { + decision_p [h_p] = (v,t) // decide on time too + h_p ← h_p + 1 + reset lockedRound_p , lockedValue_p, validRound_p and validValue_p to initial values and empty message log + StartRound(0) + } +} +``` + +**All other rules remains unchanged.** + +Back to [main document][main]. + +[main]: ./pbts_001_draft.md + +[arXiv]: https://arxiv.org/abs/1807.04938 + + + diff --git a/cometbft/v0.38/spec/consensus/proposer-based-timestamp/pbts-sysmodel_001_draft.md b/cometbft/v0.38/spec/consensus/proposer-based-timestamp/pbts-sysmodel_001_draft.md new file mode 100644 index 00000000..06f9e8ea --- /dev/null +++ b/cometbft/v0.38/spec/consensus/proposer-based-timestamp/pbts-sysmodel_001_draft.md @@ -0,0 +1,191 @@ +# Proposer-Based Time - Part I + +## System Model + +### Time and Clocks + +#### **[PBTS-CLOCK-NEWTON.0]** + +There is a reference Newtonian real-time `t` (UTC). + +Every correct validator `V` maintains a synchronized clock `C_V` that ensures: + +#### **[PBTS-CLOCK-PRECISION.0]** + +There exists a system parameter `PRECISION` such that for any two correct validators `V` and `W`, and at any real-time `t`, +`|C_V(t) - C_W(t)| < PRECISION` + + +### Message Delays + +We do not want to interfere with the timing assumptions of Tendermint consensus algorithm. +We will postulate a timing restriction, which, if satisfied, ensures that liveness is preserved. + +In general the local clock may drift from the global time. (It may progress faster, e.g., one second of clock time might take 1.005 seconds of real-time). As a result the local clock and the global clock may be measured in different time units. Usually, the message delay is measured in global clock time units. To estimate the correct local timeout precisely, we would need to estimate the clock time duration of a message delay taking into account the clock drift. For simplicity we ignore this, and directly postulate the message delay assumption in terms of local time. + + +#### **[PBTS-MSG-D.0]** + +There exists a system parameter `MSGDELAY` for message end-to-end delays **counted in clock-time**. + +> Observe that [PBTS-MSG-D.0] imposes constraints on message delays as well as on the clock. + +#### **[PBTS-MSG-FAIR.0]** + +The message end-to-end delay between a correct proposer and a correct validator (for `PROPOSE` messages) is less than `MSGDELAY`. + + +## Problem Statement + +In this section we define the properties of Tendermint consensus algorithm (cf. the [arXiv paper][arXiv]) in this new system model. + +#### **[PBTS-PROPOSE.0]** + +A proposer proposes a pair `(v,t)` of consensus value `v` and time `t`. + +> We then restrict the allowed decisions along the following lines: + +#### **[PBTS-INV-AGREEMENT.0]** + +[Agreement] No two correct validators decide on different values `v`. + +#### **[PBTS-INV-TIME-VAL.0]** + +[Time-Validity] If a correct validator decides on `t` then `t` is "OK" (we will formalize this below), even if up to `2f` validators are faulty. + +However, the properties of Tendermint consensus algorithm are of more interest with respect to the blocks, that is, what is written into a block and when. We therefore, in the following, will give the safety and liveness properties from this block-centric viewpoint. +For this, observe that the time `t` decided at consensus height `k` will be written in the block of height `k+1`, and will be supported by `2f + 1` `PRECOMMIT` messages of the same consensus round `r`. The time written in the block, we will denote by `b.time` (to distinguish it from the term `bfttime` used for median-based time). For this, it is important to have the following consensus algorithm property: + +#### **[PBTS-INV-TIME-AGR.0]** + +[Time-Agreement] If two correct validators decide in the same round, then they decide on the same `t`. + +#### **[PBTS-DECISION-ROUND.0]** + +Note that the relation between consensus decisions, on the one hand, and blocks, on the other hand, is not immediate; in particular if we consider time: In the proposed solution, +as validators may decide in different rounds, they may decide on different times. +The proposer of the next block, may pick a commit (at least `2f + 1` `PRECOMMIT` messages from one round), and thus it picks a decision round that is going to become "canonic". +As a result, the proposer implicitly has a choice of one of the times that belong to rounds in which validators decided. Observe that this choice was implicitly the case already in the median-based `bfttime`. +However, as most consensus instances terminate within one round on the Cosmos hub, this is hardly ever observed in practice. + + + +Finally, observe that the agreement ([Agreement] and [Time-Agreement]) properties are based on the Cosmos security model [CMBC-FM-2THIRDS.0][CMBC-FM-2THIRDS-link] of more than 2/3 correct validators, while [Time-Validity] is based on more than 1/3 correct validators. + +### SAFETY + +Here we will provide specifications that relate local time to block time. However, since we do not assume (by now) that local time is linked to real-time, these specifications also do not provide a relation between block time and real-time. Such properties are given [later](#real-time-safety). + +For a correct validator `V`, let `beginConsensus(V,k)` be the local time when it sets its height to `k`, and let `endConsensus(V,k)` be the time when it sets its height to `k + 1`. + +Let + +- `beginConsensus(k)` be the minimum over `beginConsensus(V,k)`, and +- `last-beginConsensus(k)` be the maximum over `beginConsensus(V,k)`, and +- `endConsensus(k)` the maximum over `endConsensus(V,k)` + +for all correct validators `V`. + +> Observe that `beginConsensus(k) <= last-beginConsensus(k)` and if local clocks are monotonic, then `last-beginConsensus(k) <= endConsensus(k)`. + +#### **[PBTS-CLOCK-GROW.0]** + +We assume that during one consensus instance, local clocks are not set back, in particular for each correct validator `V` and each height `k`, we have `beginConsensus(V,k) < endConsensus(V,k)`. + + +#### **[PBTS-CONSENSUS-TIME-VALID.0]** + +If + +- there is a valid commit `c` for height `k`, and +- `c` contains a `PRECOMMIT` message by at least one correct validator, + +then the time `b.time` in the block `b` that is signed by `c` satisfies + +- `beginConsensus(k) - PRECISION <= b.time < endConsensus(k) + PRECISION + MSGDELAY`. + + +> [PBTS-CONSENSUS-TIME-VALID.0] is based on an analysis where the proposer is faulty (and does does not count towards `beginConsensus(k)` and `endConsensus(k)`), and we estimate the times at which correct validators receive and `accept` the `propose` message. If the proposer is correct we obtain + +#### **[PBTS-CONSENSUS-LIVE-VALID-CORR-PROP.0]** + +If the proposer of round 1 is correct, and + +- [CMBC-FM-2THIRDS.0] holds for a block of height `k - 1`, and +- [PBTS-MSG-FAIR.0], and +- [PBTS-CLOCK-PRECISION.0], and +- [PBTS-CLOCK-GROW.0] (**TODO:** is that enough?) + +then eventually (within bounded time) every correct validator decides in round 1. + +#### **[PBTS-CONSENSUS-SAFE-VALID-CORR-PROP.0]** + +If the proposer of round 1 is correct, and + +- [CMBC-FM-2THIRDS.0] holds for a block of height `k - 1`, and +- [PBTS-MSG-FAIR.0], and +- [PBTS-CLOCK-PRECISION.0], and +- [PBTS-CLOCK-GROW.0] (**TODO:** is that enough?) + +then `beginConsensus_k <= b.time <= last-beginConsensus_k`. + + +> For the above two properties we will assume that a correct proposer `v` sends its `PROPOSAL` at its local time `beginConsensus(v,k)`. + +### LIVENESS + +If + +- [CMBC-FM-2THIRDS.0] holds for a block of height `k - 1`, and +- [PBTS-MSG-FAIR.0], +- [PBTS-CLOCK.0], and +- [PBTS-CLOCK-GROW.0] (**TODO:** is that enough?) + +then eventually there is a valid commit `c` for height `k`. + + +### REAL-TIME SAFETY + +> We want to give a property that can be exploited from the outside, that is, given a block with some time stored in it, what is the estimate at which real-time the block was generated. To do so, we need to link clock-time to real-time; which is not the case with [PBTS-CLOCK.0]. For this, we introduce the following assumption on the clocks: + +#### **[PBTS-CLOCKSYNC-EXTERNAL.0]** + +There is a system parameter `ACCURACY`, such that for all real-times `t` and all correct validators `V`, + +- `| C_V(t) - t | < ACCURACY`. + +> `ACCURACY` is not necessarily visible at the code level. The properties below just show that the smaller +its value, the closer the block time will be to real-time + +#### **[PBTS-CONSENSUS-PTIME.0]** + +LET `m` be a propose message. We consider the following two real-times `proposalTime(m)` and `propRecvTime(m)`: + +- if the proposer is correct and sends `m` at time `t`, we write `proposalTime(m)` for real-time `t`. +- if first correct validator receives `m` at time `t`, we write `propRecvTime(m)` for real-time `t`. + + +#### **[PBTS-CONSENSUS-REALTIME-VALID.0]** + +Let `b` be a block with a valid commit that contains at least one `precommit` message by a correct validator (and `proposalTime` is the time for the height/round `propose` message `m` that triggered the `precommit`). Then: + +`propRecvTime(m) - ACCURACY - PRECISION < b.time < propRecvTime(m) + ACCURACY + PRECISION + MSGDELAY` + + +#### **[PBTS-CONSENSUS-REALTIME-VALID-CORR.0]** + +Let `b` be a block with a valid commit that contains at least one `precommit` message by a correct validator (and `proposalTime` is the time for the height/round `propose` message `m` that triggered the `precommit`). Then, if the proposer is correct: + +`proposalTime(m) - ACCURACY < b.time < proposalTime(m) + ACCURACY` + +> by the algorithm at time `proposalTime(m)` the proposer fixes `m.time <- now_p(proposalTime(m))` + +> "triggered the `PRECOMMIT`" implies that the data in `m` and `b` are "matching", that is, `m` proposed the values that are actually stored in `b`. + +Back to [main document][main]. + +[main]: ./pbts_001_draft.md + +[arXiv]: https://arxiv.org/abs/1807.04938 + +[CMBC-FM-2THIRDS-link]: https://github.com/cometbft/cometbft/blob/v0.38.x/spec/light-client/verification/verification_002_draft.md#cmbc-fm-2thirds1 diff --git a/cometbft/v0.38/spec/consensus/proposer-based-timestamp/pbts_001_draft.md b/cometbft/v0.38/spec/consensus/proposer-based-timestamp/pbts_001_draft.md new file mode 100644 index 00000000..f71d7ab8 --- /dev/null +++ b/cometbft/v0.38/spec/consensus/proposer-based-timestamp/pbts_001_draft.md @@ -0,0 +1,269 @@ +# Proposer-Based Time + +## Current BFTTime + +### Description + +In CometBFT, the first version of how time is computed and stored in a block works as follows: + +- validators send their current local time as part of `precommit` messages +- upon collecting the `precommit` messages that the proposer uses to build a commit to be put in the next block, the proposer computes the `time` of the next block as the median (weighted over voting power) of the times in the `precommit` messages. + +### Analysis + +1. **Fault tolerance.** The computed median time is called [`bfttime`][bfttime] as it is indeed fault-tolerant: if **less than a third** of the validators is faulty (counted in voting power), it is guaranteed that the computed time lies between the minimum and the maximum times sent by correct validators. +1. **Effect of faulty validators.** If more than `1/2` of the voting power (which is in fact more than one third and less than two thirds of the voting power) is held by faulty validators, then the time is under total control of the faulty validators. (This is particularly challenging in the context of [lightclient][lcspec] security.) +1. **Proposer influence on block time.** The proposer of the next block has a degree of freedom in choosing the `bfttime`, since it computes the median time based on the timestamps from `precommit` messages sent by + `2f + 1` correct validators. + 1. If there are `n` different timestamps in the `precommit` messages, the proposer can use any subset of timestamps that add up to `2f + 1` + of the voting power in order to compute the median. + 1. If the validators decide in different rounds, the proposer can decide on which round the median computation is based. +1. **Liveness.** The liveness of the protocol: + 1. does not depend on clock synchronization, + 1. depends on bounded message delays. +1. **Relation to real time.** There is no clock synchronizaton, which implies that there is **no relation** between the computed block `time` and real time. +1. **Aggregate signatures.** As the `precommit` messages contain the local times, all these `precommit` messages typically differ in the time field, which **prevents** the use of aggregate signatures. + +## Suggested Proposer-Based Time + +### Outline + +An alternative approach to time has been discussed: Rather than having the validators send the time in the `precommit` messages, the proposer in the consensus algorithm sends its time in the `propose` message, and the validators locally check whether the time is OK (by comparing to their local clock). + +This proposed solution adds the requirement of having synchronized clocks, and other implicit assumptions. + +### Comparison of the Suggested Method to the Old One + +1. **Fault tolerance.** Maintained in the suggested protocol. +1. **Effect of faulty validators.** Eliminated in the suggested protocol, + that is, the block `time` can be corrupted only in the extreme case when + `>2/3` of the validators are faulty. +1. **Proposer influence on block time.** The proposer of the next block + has less freedom when choosing the block time. + 1. This scenario is eliminated in the suggested protocol, provided that there are `<1/3` faulty validators. + 1. This scenario is still there. +1. **Liveness.** The liveness of the suggested protocol: + 1. depends on the introduced assumptions on synchronized clocks (see below), + 1. still depends on the message delays (unavoidable). +1. **Relation to real time.** We formalize clock synchronization, and obtain a **well-defined relation** between the block `time` and real time. +1. **Aggregate signatures.** The `precommit` messages free of time, which **allows** for aggregate signatures. + +### Protocol Overview + +#### Proposed Time + +We assume that the field `proposal` in the `PROPOSE` message is a pair `(v, time)`, of the proposed consensus value `v` and the proposed time `time`. + +#### Reception Step + +In the reception step at node `p` at local time `now_p`, upon receiving a message `m`: + +- **if** the message `m` is of type `PROPOSE` and satisfies `now_p - PRECISION < m.time < now_p + PRECISION + MSGDELAY`, then mark the message as `timely`. +(`PRECISION` and `MSGDELAY` being system parameters, see [below](#safety-and-liveness)) + +> after the presentation in the dev session, we realized that different semantics for the reception step is closer aligned to the implementation. Instead of dropping propose messages, we keep all of them, and mark timely ones. + +#### Processing Step + +- Start round + + + + + + + + + + + + +
arXiv paperProposer-based time
+ +```go +function StartRound(round) { + round_p ← round + step_p ← propose + if proposer(h_p, round_p) = p { + + + if validValue_p != nil { + + proposal ← validValue_p + } else { + + proposal ← getValue() + } + broadcast ⟨PROPOSAL, h_p, round_p, proposal, validRound_p⟩ + } else { + schedule OnTimeoutPropose(h_p,round_p) to + be executed after timeoutPropose(round_p) + } +} +``` + + + +```go +function StartRound(round) { + round_p ← round + step_p ← propose + if proposer(h_p, round_p) = p { + // new wait condition + wait until now_p > block time of block h_p - 1 + if validValue_p != nil { + // add "now_p" + proposal ← (validValue_p, now_p) + } else { + // add "now_p" + proposal ← (getValue(), now_p) + } + broadcast ⟨PROPOSAL, h_p, round_p, proposal, validRound_p⟩ + } else { + schedule OnTimeoutPropose(h_p,round_p) to + be executed after timeoutPropose(round_p) + } +} +``` + +
+ +- Rule on lines 28-35 + + + + + + + + + + + + +
arXiv paperProposer-based time
+ +```go +upon timely(⟨PROPOSAL, h_p, round_p, v, vr⟩) + from proposer(h_p, round_p) + AND 2f + 1 ⟨PREVOTE, h_p, vr, id(v)⟩ +while step_p = propose ∧ (vr ≥ 0 ∧ vr < round_p) do { + if valid(v) ∧ (lockedRound_p ≤ vr ∨ lockedValue_p = v) { + + broadcast ⟨PREVOTE, h_p, round_p, id(v)⟩ + } else { + broadcast ⟨PREVOTE, hp, round_p, nil⟩ + } +} +``` + + + +```go +upon timely(⟨PROPOSAL, h_p, round_p, (v, tprop), vr⟩) + from proposer(h_p, round_p) + AND 2f + 1 ⟨PREVOTE, h_p, vr, id(v, tvote)⟩ + while step_p = propose ∧ (vr ≥ 0 ∧ vr < round_p) do { + if valid(v) ∧ (lockedRound_p ≤ vr ∨ lockedValue_p = v) { + // send hash of v and tprop in PREVOTE message + broadcast ⟨PREVOTE, h_p, round_p, id(v, tprop)⟩ + } else { + broadcast ⟨PREVOTE, hp, round_p, nil⟩ + } + } +``` + +
+ +- Rule on lines 49-54 + + + + + + + + + + + + +
arXiv paperProposer-based time
+ +```go +upon ⟨PROPOSAL, h_p, r, v, ∗⟩ from proposer(h_p, r) + AND 2f + 1 ⟨PRECOMMIT, h_p, r, id(v)⟩ + while decisionp[h_p] = nil do { + if valid(v) { + + decision_p [h_p] = v + h_p ← h_p + 1 + reset lockedRound_p , lockedValue_p, validRound_p and + validValue_p to initial values and empty message log + StartRound(0) + } + } +``` + + + +```go +upon ⟨PROPOSAL, h_p, r, (v,t), ∗⟩ from proposer(h_p, r) + AND 2f + 1 ⟨PRECOMMIT, h_p, r, id(v,t)⟩ + while decisionp[h_p] = nil do { + if valid(v) { + // decide on time too + decision_p [h_p] = (v,t) + h_p ← h_p + 1 + reset lockedRound_p , lockedValue_p, validRound_p and + validValue_p to initial values and empty message log + StartRound(0) + } + } +``` + +
+ +- Other rules are extended in a similar way, or remain unchanged + +### Property Overview + +#### Safety and Liveness + +For safety (Point 1, Point 2, Point 3i) and liveness (Point 4) we need +the following assumptions: + +- There exists a system parameter `PRECISION` such that for any two correct validators `V` and `W`, and at any real-time `t`, their local times `C_V(t)` and `C_W(t)` differ by less than `PRECISION` time units, +i.e., `|C_V(t) - C_W(t)| < PRECISION` +- The message end-to-end delay between a correct proposer and a correct validator (for `PROPOSE` messages) is less than `MSGDELAY`. + +#### Relation to Real-Time + +For analyzing real-time safety (Point 5), we use a system parameter `ACCURACY`, such that for all real-times `t` and all correct validators `V`, we have `| C_V(t) - t | < ACCURACY`. + +> `ACCURACY` is not necessarily visible at the code level. We might even view `ACCURACY` as variable over time. The smaller it is during a consensus instance, the closer the block time will be to real-time. +> +> Note that `PRECISION` and `MSGDELAY` show up in the code. + +### Detailed Specification + +This specification describes the changes needed to be done to the Tendermint consensus algorithm as described in the [arXiv paper][arXiv] and the simplified specification in [TLA+][tlatender], and makes precise the underlying assumptions and the required properties. + +- [Part I - System Model and Properties][sysmodel] +- [Part II - Protocol specification][algorithm] +- [TLA+ Specification][proposertla] + +[arXiv]: https://arxiv.org/abs/1807.04938 + +[tlatender]: ../../light-client/accountability/README.md + +[bfttime]: ../bft-time.md + +[lcspec]: ../../light-client/README.md + +[algorithm]: ./pbts-algorithm_001_draft.md + +[sysmodel]: ./pbts-sysmodel_001_draft.md + + +[proposertla]: ./tla/TendermintPBT_001_draft.tla diff --git a/cometbft/v0.38/spec/consensus/proposer-based-timestamp/tla/TendermintPBT_001_draft.tla b/cometbft/v0.38/spec/consensus/proposer-based-timestamp/tla/TendermintPBT_001_draft.tla new file mode 100644 index 00000000..d8524540 --- /dev/null +++ b/cometbft/v0.38/spec/consensus/proposer-based-timestamp/tla/TendermintPBT_001_draft.tla @@ -0,0 +1,597 @@ +-------------------- MODULE TendermintPBT_001_draft --------------------------- +(* + A TLA+ specification of a simplified Tendermint consensus algorithm, with added clocks + and proposer-based timestamps. This TLA+ specification extends and modifies + the Tendermint TLA+ specification for fork accountability: + https://github.com/cometbft/cometbft/blob/v0.38.x/spec/light-client/accountability/TendermintAcc_004_draft.tla + + * Version 1. A preliminary specification. + + Zarko Milosevic, Igor Konnov, Informal Systems, 2019-2020. + Ilina Stoilkovska, Josef Widder, Informal Systems, 2021. + *) + +EXTENDS Integers, FiniteSets + +(********************* PROTOCOL PARAMETERS **********************************) +CONSTANTS + Corr, \* the set of correct processes + Faulty, \* the set of Byzantine processes, may be empty + N, \* the total number of processes: correct, defective, and Byzantine + T, \* an upper bound on the number of Byzantine processes + ValidValues, \* the set of valid values, proposed both by correct and faulty + InvalidValues, \* the set of invalid values, never proposed by the correct ones + MaxRound, \* the maximal round number + MaxTimestamp, \* the maximal value of the clock tick + Delay, \* message delay + Precision, \* clock precision: the maximal difference between two local clocks + Accuracy, \* clock accuracy: the maximal difference between a local clock and the real time + Proposer, \* the proposer function from 0..NRounds to 1..N + ClockDrift \* is there clock drift between the local clocks and the global clock + +ASSUME(N = Cardinality(Corr \union Faulty)) + +(*************************** DEFINITIONS ************************************) +AllProcs == Corr \union Faulty \* the set of all processes +Rounds == 0..MaxRound \* the set of potential rounds +Timestamps == 0..MaxTimestamp \* the set of clock ticks +NilRound == -1 \* a special value to denote a nil round, outside of Rounds +NilTimestamp == -1 \* a special value to denote a nil timestamp, outside of Ticks +RoundsOrNil == Rounds \union {NilRound} +Values == ValidValues \union InvalidValues \* the set of all values +NilValue == "None" \* a special value for a nil round, outside of Values +Proposals == Values \X Timestamps +NilProposal == <> +ValuesOrNil == Values \union {NilValue} +Decisions == Values \X Timestamps \X Rounds +NilDecision == <> + + +\* a value hash is modeled as identity +Id(v) == v + +\* The validity predicate +IsValid(v) == v \in ValidValues + +\* the two thresholds that are used in the algorithm +THRESHOLD1 == T + 1 \* at least one process is not faulty +THRESHOLD2 == 2 * T + 1 \* a quorum when having N > 3 * T + +Min(S) == CHOOSE x \in S : \A y \in S : x <= y + +Max(S) == CHOOSE x \in S : \A y \in S : y <= x + +(********************* TYPE ANNOTATIONS FOR APALACHE **************************) +\* the operator for type annotations +a <: b == a + +\* the type of message records +MT == [type |-> STRING, src |-> STRING, round |-> Int, + proposal |-> <>, validRound |-> Int, id |-> <>] + +RP == <> + +\* a type annotation for a message +AsMsg(m) == m <: MT +\* a type annotation for a set of messages +SetOfMsgs(S) == S <: {MT} +\* a type annotation for an empty set of messages +EmptyMsgSet == SetOfMsgs({}) + +SetOfRcvProp(S) == S <: {RP} +EmptyRcvProp == SetOfRcvProp({}) + +SetOfProc(S) == S <: {STRING} +EmptyProcSet == SetOfProc({}) + +(********************* PROTOCOL STATE VARIABLES ******************************) +VARIABLES + round, \* a process round number: Corr -> Rounds + localClock, \* a process local clock: Corr -> Ticks + realTime, \* a reference Newtonian real time + step, \* a process step: Corr -> { "PROPOSE", "PREVOTE", "PRECOMMIT", "DECIDED" } + decision, \* process decision: Corr -> ValuesOrNil + lockedValue, \* a locked value: Corr -> ValuesOrNil + lockedRound, \* a locked round: Corr -> RoundsOrNil + validValue, \* a valid value: Corr -> ValuesOrNil + validRound \* a valid round: Corr -> RoundsOrNil + +\* book-keeping variables +VARIABLES + msgsPropose, \* PROPOSE messages broadcast in the system, Rounds -> Messages + msgsPrevote, \* PREVOTE messages broadcast in the system, Rounds -> Messages + msgsPrecommit, \* PRECOMMIT messages broadcast in the system, Rounds -> Messages + receivedTimelyProposal, \* used to keep track when a process receives a timely PROPOSAL message, {<>} + inspectedProposal, \* used to keep track when a process tries to receive a message, [Rounds -> <>] + evidence, \* the messages that were used by the correct processes to make transitions + action, \* we use this variable to see which action was taken + beginConsensus, \* the minimum of the local clocks in the initial state, Int + endConsensus, \* the local time when a decision is made, [Corr -> Int] + lastBeginConsensus, \* the maximum of the local clocks in the initial state, Int + proposalTime, \* the real time when a proposer proposes in a round, [Rounds -> Int] + proposalReceivedTime \* the real time when a correct process first receives a proposal message in a round, [Rounds -> Int] + +(* to see a type invariant, check TendermintAccInv3 *) + +\* a handy definition used in UNCHANGED +vars == <> + +(********************* PROTOCOL INITIALIZATION ******************************) +FaultyProposals(r) == + SetOfMsgs([type: {"PROPOSAL"}, src: Faulty, + round: {r}, proposal: Proposals, validRound: RoundsOrNil]) + +AllFaultyProposals == + SetOfMsgs([type: {"PROPOSAL"}, src: Faulty, + round: Rounds, proposal: Proposals, validRound: RoundsOrNil]) + +FaultyPrevotes(r) == + SetOfMsgs([type: {"PREVOTE"}, src: Faulty, round: {r}, id: Proposals]) + +AllFaultyPrevotes == + SetOfMsgs([type: {"PREVOTE"}, src: Faulty, round: Rounds, id: Proposals]) + +FaultyPrecommits(r) == + SetOfMsgs([type: {"PRECOMMIT"}, src: Faulty, round: {r}, id: Proposals]) + +AllFaultyPrecommits == + SetOfMsgs([type: {"PRECOMMIT"}, src: Faulty, round: Rounds, id: Proposals]) + +AllProposals == + SetOfMsgs([type: {"PROPOSAL"}, src: AllProcs, + round: Rounds, proposal: Proposals, validRound: RoundsOrNil]) + +RoundProposals(r) == + SetOfMsgs([type: {"PROPOSAL"}, src: AllProcs, + round: {r}, proposal: Proposals, validRound: RoundsOrNil]) + +BenignRoundsInMessages(msgfun) == + \* the message function never contains a message for a wrong round + \A r \in Rounds: + \A m \in msgfun[r]: + r = m.round + +\* The initial states of the protocol. Some faults can be in the system already. +Init == + /\ round = [p \in Corr |-> 0] + /\ \/ /\ ~ClockDrift + /\ localClock \in [Corr -> 0..Accuracy] + \/ /\ ClockDrift + /\ localClock = [p \in Corr |-> 0] + /\ realTime = 0 + /\ step = [p \in Corr |-> "PROPOSE"] + /\ decision = [p \in Corr |-> NilDecision] + /\ lockedValue = [p \in Corr |-> NilValue] + /\ lockedRound = [p \in Corr |-> NilRound] + /\ validValue = [p \in Corr |-> NilValue] + /\ validRound = [p \in Corr |-> NilRound] + /\ msgsPropose \in [Rounds -> SUBSET AllFaultyProposals] + /\ msgsPrevote \in [Rounds -> SUBSET AllFaultyPrevotes] + /\ msgsPrecommit \in [Rounds -> SUBSET AllFaultyPrecommits] + /\ receivedTimelyProposal = EmptyRcvProp + /\ inspectedProposal = [r \in Rounds |-> EmptyProcSet] + /\ BenignRoundsInMessages(msgsPropose) + /\ BenignRoundsInMessages(msgsPrevote) + /\ BenignRoundsInMessages(msgsPrecommit) + /\ evidence = EmptyMsgSet + /\ action' = "Init" + /\ beginConsensus = Min({localClock[p] : p \in Corr}) + /\ endConsensus = [p \in Corr |-> NilTimestamp] + /\ lastBeginConsensus = Max({localClock[p] : p \in Corr}) + /\ proposalTime = [r \in Rounds |-> NilTimestamp] + /\ proposalReceivedTime = [r \in Rounds |-> NilTimestamp] + +(************************ MESSAGE PASSING ********************************) +BroadcastProposal(pSrc, pRound, pProposal, pValidRound) == + LET newMsg == + AsMsg([type |-> "PROPOSAL", src |-> pSrc, round |-> pRound, + proposal |-> pProposal, validRound |-> pValidRound]) + IN + msgsPropose' = [msgsPropose EXCEPT ![pRound] = msgsPropose[pRound] \union {newMsg}] + +BroadcastPrevote(pSrc, pRound, pId) == + LET newMsg == AsMsg([type |-> "PREVOTE", + src |-> pSrc, round |-> pRound, id |-> pId]) + IN + msgsPrevote' = [msgsPrevote EXCEPT ![pRound] = msgsPrevote[pRound] \union {newMsg}] + +BroadcastPrecommit(pSrc, pRound, pId) == + LET newMsg == AsMsg([type |-> "PRECOMMIT", + src |-> pSrc, round |-> pRound, id |-> pId]) + IN + msgsPrecommit' = [msgsPrecommit EXCEPT ![pRound] = msgsPrecommit[pRound] \union {newMsg}] + + +(***************************** TIME **************************************) + +\* [PBTS-CLOCK-PRECISION.0] +SynchronizedLocalClocks == + \A p \in Corr : \A q \in Corr : + p /= q => + \/ /\ localClock[p] >= localClock[q] + /\ localClock[p] - localClock[q] < Precision + \/ /\ localClock[p] < localClock[q] + /\ localClock[q] - localClock[p] < Precision + +\* [PBTS-PROPOSE.0] +Proposal(v, t) == + <> + +\* [PBTS-DECISION-ROUND.0] +Decision(v, t, r) == + <> + +(**************** MESSAGE PROCESSING TRANSITIONS *************************) +\* lines 12-13 +StartRound(p, r) == + /\ step[p] /= "DECIDED" \* a decided process does not participate in consensus + /\ round' = [round EXCEPT ![p] = r] + /\ step' = [step EXCEPT ![p] = "PROPOSE"] + +\* lines 14-19, a proposal may be sent later +InsertProposal(p) == + LET r == round[p] IN + /\ p = Proposer[r] + /\ step[p] = "PROPOSE" + \* if the proposer is sending a proposal, then there are no other proposals + \* by the correct processes for the same round + /\ \A m \in msgsPropose[r]: m.src /= p + /\ \E v \in ValidValues: + LET proposal == IF validValue[p] /= NilValue + THEN Proposal(validValue[p], localClock[p]) + ELSE Proposal(v, localClock[p]) IN + + /\ BroadcastProposal(p, round[p], proposal, validRound[p]) + /\ proposalTime' = [proposalTime EXCEPT ![r] = realTime] + /\ UNCHANGED <> + /\ action' = "InsertProposal" + +\* a new action used to filter messages that are not on time +\* [PBTS-RECEPTION-STEP.0] +ReceiveProposal(p) == + \E v \in Values, t \in Timestamps: + /\ LET r == round[p] IN + LET msg == + AsMsg([type |-> "PROPOSAL", src |-> Proposer[round[p]], + round |-> round[p], proposal |-> Proposal(v, t), validRound |-> NilRound]) IN + /\ msg \in msgsPropose[round[p]] + /\ p \notin inspectedProposal[r] + /\ <> \notin receivedTimelyProposal + /\ inspectedProposal' = [inspectedProposal EXCEPT ![r] = @ \union {p}] + /\ \/ /\ localClock[p] - Precision < t + /\ t < localClock[p] + Precision + Delay + /\ receivedTimelyProposal' = receivedTimelyProposal \union {<>} + /\ \/ /\ proposalReceivedTime[r] = NilTimestamp + /\ proposalReceivedTime' = [proposalReceivedTime EXCEPT ![r] = realTime] + \/ /\ proposalReceivedTime[r] /= NilTimestamp + /\ UNCHANGED proposalReceivedTime + \/ /\ \/ localClock[p] - Precision >= t + \/ t >= localClock[p] + Precision + Delay + /\ UNCHANGED <> + /\ UNCHANGED <> + /\ action' = "ReceiveProposal" + +\* lines 22-27 +UponProposalInPropose(p) == + \E v \in Values, t \in Timestamps: + /\ step[p] = "PROPOSE" (* line 22 *) + /\ LET msg == + AsMsg([type |-> "PROPOSAL", src |-> Proposer[round[p]], + round |-> round[p], proposal |-> Proposal(v, t), validRound |-> NilRound]) IN + /\ <> \in receivedTimelyProposal \* updated line 22 + /\ evidence' = {msg} \union evidence + /\ LET mid == (* line 23 *) + IF IsValid(v) /\ (lockedRound[p] = NilRound \/ lockedValue[p] = v) + THEN Id(Proposal(v, t)) + ELSE NilProposal + IN + BroadcastPrevote(p, round[p], mid) \* lines 24-26 + /\ step' = [step EXCEPT ![p] = "PREVOTE"] + /\ UNCHANGED <> + /\ action' = "UponProposalInPropose" + +\* lines 28-33 +\* [PBTS-ALG-OLD-PREVOTE.0] +UponProposalInProposeAndPrevote(p) == + \E v \in Values, t1 \in Timestamps, t2 \in Timestamps, vr \in Rounds: + /\ step[p] = "PROPOSE" /\ 0 <= vr /\ vr < round[p] \* line 28, the while part + /\ LET msg == + AsMsg([type |-> "PROPOSAL", src |-> Proposer[round[p]], + round |-> round[p], proposal |-> Proposal(v, t1), validRound |-> vr]) + IN + /\ <> \in receivedTimelyProposal \* updated line 28 + /\ LET PV == { m \in msgsPrevote[vr]: m.id = Id(Proposal(v, t2)) } IN + /\ Cardinality(PV) >= THRESHOLD2 \* line 28 + /\ evidence' = PV \union {msg} \union evidence + /\ LET mid == (* line 29 *) + IF IsValid(v) /\ (lockedRound[p] <= vr \/ lockedValue[p] = v) + THEN Id(Proposal(v, t1)) + ELSE NilProposal + IN + BroadcastPrevote(p, round[p], mid) \* lines 24-26 + /\ step' = [step EXCEPT ![p] = "PREVOTE"] + /\ UNCHANGED <> + /\ action' = "UponProposalInProposeAndPrevote" + + \* lines 34-35 + lines 61-64 (onTimeoutPrevote) +UponQuorumOfPrevotesAny(p) == + /\ step[p] = "PREVOTE" \* line 34 and 61 + /\ \E MyEvidence \in SUBSET msgsPrevote[round[p]]: + \* find the unique voters in the evidence + LET Voters == { m.src: m \in MyEvidence } IN + \* compare the number of the unique voters against the threshold + /\ Cardinality(Voters) >= THRESHOLD2 \* line 34 + /\ evidence' = MyEvidence \union evidence + /\ BroadcastPrecommit(p, round[p], NilProposal) + /\ step' = [step EXCEPT ![p] = "PRECOMMIT"] + /\ UNCHANGED <> + /\ action' = "UponQuorumOfPrevotesAny" + +\* lines 36-46 +\* [PBTS-ALG-NEW-PREVOTE.0] +UponProposalInPrevoteOrCommitAndPrevote(p) == + \E v \in ValidValues, t \in Timestamps, vr \in RoundsOrNil: + /\ step[p] \in {"PREVOTE", "PRECOMMIT"} \* line 36 + /\ LET msg == + AsMsg([type |-> "PROPOSAL", src |-> Proposer[round[p]], + round |-> round[p], proposal |-> Proposal(v, t), validRound |-> vr]) IN + /\ <> \in receivedTimelyProposal \* updated line 36 + /\ LET PV == { m \in msgsPrevote[round[p]]: m.id = Id(Proposal(v, t)) } IN + /\ Cardinality(PV) >= THRESHOLD2 \* line 36 + /\ evidence' = PV \union {msg} \union evidence + /\ IF step[p] = "PREVOTE" + THEN \* lines 38-41: + /\ lockedValue' = [lockedValue EXCEPT ![p] = v] + /\ lockedRound' = [lockedRound EXCEPT ![p] = round[p]] + /\ BroadcastPrecommit(p, round[p], Id(Proposal(v, t))) + /\ step' = [step EXCEPT ![p] = "PRECOMMIT"] + ELSE + UNCHANGED <> + \* lines 42-43 + /\ validValue' = [validValue EXCEPT ![p] = v] + /\ validRound' = [validRound EXCEPT ![p] = round[p]] + /\ UNCHANGED <> + /\ action' = "UponProposalInPrevoteOrCommitAndPrevote" + +\* lines 47-48 + 65-67 (onTimeoutPrecommit) +UponQuorumOfPrecommitsAny(p) == + /\ \E MyEvidence \in SUBSET msgsPrecommit[round[p]]: + \* find the unique committers in the evidence + LET Committers == { m.src: m \in MyEvidence } IN + \* compare the number of the unique committers against the threshold + /\ Cardinality(Committers) >= THRESHOLD2 \* line 47 + /\ evidence' = MyEvidence \union evidence + /\ round[p] + 1 \in Rounds + /\ StartRound(p, round[p] + 1) + /\ UNCHANGED <> + /\ action' = "UponQuorumOfPrecommitsAny" + +\* lines 49-54 +\* [PBTS-ALG-DECIDE.0] +UponProposalInPrecommitNoDecision(p) == + /\ decision[p] = NilDecision \* line 49 + /\ \E v \in ValidValues, t \in Timestamps (* line 50*) , r \in Rounds, vr \in RoundsOrNil: + /\ LET msg == AsMsg([type |-> "PROPOSAL", src |-> Proposer[r], + round |-> r, proposal |-> Proposal(v, t), validRound |-> vr]) IN + /\ msg \in msgsPropose[r] \* line 49 + /\ p \in inspectedProposal[r] + /\ LET PV == { m \in msgsPrecommit[r]: m.id = Id(Proposal(v, t)) } IN + /\ Cardinality(PV) >= THRESHOLD2 \* line 49 + /\ evidence' = PV \union {msg} \union evidence + /\ decision' = [decision EXCEPT ![p] = Decision(v, t, round[p])] \* update the decision, line 51 + \* The original algorithm does not have 'DECIDED', but it increments the height. + \* We introduced 'DECIDED' here to prevent the process from changing its decision. + /\ endConsensus' = [endConsensus EXCEPT ![p] = localClock[p]] + /\ step' = [step EXCEPT ![p] = "DECIDED"] + /\ UNCHANGED <> + /\ action' = "UponProposalInPrecommitNoDecision" + +\* the actions below are not essential for safety, but added for completeness + +\* lines 20-21 + 57-60 +OnTimeoutPropose(p) == + /\ step[p] = "PROPOSE" + /\ p /= Proposer[round[p]] + /\ BroadcastPrevote(p, round[p], NilProposal) + /\ step' = [step EXCEPT ![p] = "PREVOTE"] + /\ UNCHANGED <> + /\ action' = "OnTimeoutPropose" + +\* lines 44-46 +OnQuorumOfNilPrevotes(p) == + /\ step[p] = "PREVOTE" + /\ LET PV == { m \in msgsPrevote[round[p]]: m.id = Id(NilProposal) } IN + /\ Cardinality(PV) >= THRESHOLD2 \* line 36 + /\ evidence' = PV \union evidence + /\ BroadcastPrecommit(p, round[p], Id(NilProposal)) + /\ step' = [step EXCEPT ![p] = "PRECOMMIT"] + /\ UNCHANGED <> + /\ action' = "OnQuorumOfNilPrevotes" + +\* lines 55-56 +OnRoundCatchup(p) == + \E r \in {rr \in Rounds: rr > round[p]}: + LET RoundMsgs == msgsPropose[r] \union msgsPrevote[r] \union msgsPrecommit[r] IN + \E MyEvidence \in SUBSET RoundMsgs: + LET Faster == { m.src: m \in MyEvidence } IN + /\ Cardinality(Faster) >= THRESHOLD1 + /\ evidence' = MyEvidence \union evidence + /\ StartRound(p, r) + /\ UNCHANGED <> + /\ action' = "OnRoundCatchup" + + +(********************* PROTOCOL TRANSITIONS ******************************) +\* advance the global clock +AdvanceRealTime == + /\ realTime < MaxTimestamp + /\ realTime' = realTime + 1 + /\ \/ /\ ~ClockDrift + /\ localClock' = [p \in Corr |-> localClock[p] + 1] + \/ /\ ClockDrift + /\ UNCHANGED localClock + /\ UNCHANGED <> + /\ action' = "AdvanceRealTime" + +\* advance the local clock of node p +AdvanceLocalClock(p) == + /\ localClock[p] < MaxTimestamp + /\ localClock' = [localClock EXCEPT ![p] = @ + 1] + /\ UNCHANGED <> + /\ action' = "AdvanceLocalClock" + +\* process timely messages +MessageProcessing(p) == + \* start round + \/ InsertProposal(p) + \* reception step + \/ ReceiveProposal(p) + \* processing step + \/ UponProposalInPropose(p) + \/ UponProposalInProposeAndPrevote(p) + \/ UponQuorumOfPrevotesAny(p) + \/ UponProposalInPrevoteOrCommitAndPrevote(p) + \/ UponQuorumOfPrecommitsAny(p) + \/ UponProposalInPrecommitNoDecision(p) + \* the actions below are not essential for safety, but added for completeness + \/ OnTimeoutPropose(p) + \/ OnQuorumOfNilPrevotes(p) + \/ OnRoundCatchup(p) + +(* + * A system transition. In this specificatiom, the system may eventually deadlock, + * e.g., when all processes decide. This is expected behavior, as we focus on safety. + *) +Next == + \/ AdvanceRealTime + \/ /\ ClockDrift + /\ \E p \in Corr: AdvanceLocalClock(p) + \/ /\ SynchronizedLocalClocks + /\ \E p \in Corr: MessageProcessing(p) + +----------------------------------------------------------------------------- + +(*************************** INVARIANTS *************************************) + +\* [PBTS-INV-AGREEMENT.0] +AgreementOnValue == + \A p, q \in Corr: + /\ decision[p] /= NilDecision + /\ decision[q] /= NilDecision + => \E v \in ValidValues, t1 \in Timestamps, t2 \in Timestamps, r1 \in Rounds, r2 \in Rounds : + /\ decision[p] = Decision(v, t1, r1) + /\ decision[q] = Decision(v, t2, r2) + +\* [PBTS-INV-TIME-AGR.0] +AgreementOnTime == + \A p, q \in Corr: + \A v1 \in ValidValues, v2 \in ValidValues, t1 \in Timestamps, t2 \in Timestamps, r \in Rounds : + /\ decision[p] = Decision(v1, t1, r) + /\ decision[q] = Decision(v2, t2, r) + => t1 = t2 + +\* [PBTS-CONSENSUS-TIME-VALID.0] +ConsensusTimeValid == + \A p \in Corr, t \in Timestamps : + \* if a process decides on v and t + (\E v \in ValidValues, r \in Rounds : decision[p] = Decision(v, t, r)) + \* then + => /\ beginConsensus - Precision <= t + /\ t < endConsensus[p] + Precision + Delay + +\* [PBTS-CONSENSUS-SAFE-VALID-CORR-PROP.0] +ConsensusSafeValidCorrProp == + \A v \in ValidValues, t \in Timestamps : + \* if the proposer in the first round is correct + (/\ Proposer[0] \in Corr + \* and there exists a process that decided on v, t + /\ \E p \in Corr, r \in Rounds : decision[p] = Decision(v, t, r)) + \* then t is between the minimal and maximal initial local time + => /\ beginConsensus <= t + /\ t <= lastBeginConsensus + +\* [PBTS-CONSENSUS-REALTIME-VALID-CORR.0] +ConsensusRealTimeValidCorr == + \A t \in Timestamps, r \in Rounds : + (/\ \E p \in Corr, v \in ValidValues : decision[p] = Decision(v, t, r) + /\ proposalTime[r] /= NilTimestamp) + => /\ proposalTime[r] - Accuracy < t + /\ t < proposalTime[r] + Accuracy + +\* [PBTS-CONSENSUS-REALTIME-VALID.0] +ConsensusRealTimeValid == + \A t \in Timestamps, r \in Rounds : + (\E p \in Corr, v \in ValidValues : decision[p] = Decision(v, t, r)) + => /\ proposalReceivedTime[r] - Accuracy - Precision < t + /\ t < proposalReceivedTime[r] + Accuracy + Precision + Delay + +\* [PBTS-MSG-FAIR.0] +BoundedDelay == + \A r \in Rounds : + (/\ proposalTime[r] /= NilTimestamp + /\ proposalTime[r] + Delay < realTime) + => inspectedProposal[r] = Corr + +\* [PBTS-CONSENSUS-TIME-LIVE.0] +ConsensusTimeLive == + \A r \in Rounds, p \in Corr : + (/\ proposalTime[r] /= NilTimestamp + /\ proposalTime[r] + Delay < realTime + /\ Proposer[r] \in Corr + /\ round[p] <= r) + => \E msg \in RoundProposals(r) : <> \in receivedTimelyProposal + +\* a conjunction of all invariants +Inv == + /\ AgreementOnValue + /\ AgreementOnTime + /\ ConsensusTimeValid + /\ ConsensusSafeValidCorrProp + /\ ConsensusRealTimeValid + /\ ConsensusRealTimeValidCorr + /\ BoundedDelay + +Liveness == + ConsensusTimeLive + +============================================================================= diff --git a/cometbft/v0.38/spec/core/Data_structures.mdx b/cometbft/v0.38/spec/core/Data_structures.mdx new file mode 100644 index 00000000..ecd449ad --- /dev/null +++ b/cometbft/v0.38/spec/core/Data_structures.mdx @@ -0,0 +1,501 @@ +--- +order: 1 +--- + +# Data Structures + +Here we describe the data structures in the CometBFT blockchain and the rules for validating them. + +The CometBFT blockchain consists of a short list of data types: + +- [Data Structures](#data-structures) + - [Block](#block) + - [Execution](#execution) + - [Header](#header) + - [Version](#version) + - [BlockID](#blockid) + - [PartSetHeader](#partsetheader) + - [Part](#part) + - [Time](#time) + - [Data](#data) + - [Commit](#commit) + - [CommitSig](#commitsig) + - [BlockIDFlag](#blockidflag) + - [Vote](#vote) + - [CanonicalVote](#canonicalvote) + - [Proposal](#proposal) + - [SignedMsgType](#signedmsgtype) + - [Signature](#signature) + - [EvidenceList](#evidencelist) + - [Evidence](#evidence) + - [DuplicateVoteEvidence](#duplicatevoteevidence) + - [LightClientAttackEvidence](#lightclientattackevidence) + - [LightBlock](#lightblock) + - [SignedHeader](#signedheader) + - [ValidatorSet](#validatorset) + - [Validator](#validator) + - [Address](#address) + - [ConsensusParams](#consensusparams) + - [BlockParams](#blockparams) + - [EvidenceParams](#evidenceparams) + - [ValidatorParams](#validatorparams) + - [VersionParams](#versionparams) + - [Proof](#proof) + + +## Block + +A block consists of a header, transactions, votes (the commit), +and a list of evidence of malfeasance (ie. signing conflicting votes). + +| Name | Type | Description | Validation | +|--------|-------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------| +| Header | [Header](#header) | Header corresponding to the block. This field contains information used throughout consensus and other areas of the protocol. To find out what it contains, visit [header](#header) | Must adhere to the validation rules of [header](#header) | +| Data | [Data](#data) | Data contains a list of transactions. The contents of the transaction is unknown to CometBFT. | This field can be empty or populated, but no validation is performed. Applications can perform validation on individual transactions prior to block creation using [checkTx](https://github.com/cometbft/cometbft/blob/v0.38.x/spec/abci/abci%2B%2B_methods.md#checktx). +| Evidence | [EvidenceList](#evidencelist) | Evidence contains a list of infractions committed by validators. | Can be empty, but when populated the validations rules from [evidenceList](#evidencelist) apply | +| LastCommit | [Commit](#commit) | `LastCommit` includes one vote for every validator. All votes must either be for the previous block, nil or absent. If a vote is for the previous block it must have a valid signature from the corresponding validator. The sum of the voting power of the validators that voted must be greater than 2/3 of the total voting power of the complete validator set. The number of votes in a commit is limited to 10000 (see `types.MaxVotesCount`). | Must be empty for the initial height and must adhere to the validation rules of [commit](#commit). | + +## Execution + +Once a block is validated, it can be executed against the state. + +The state follows this recursive equation: + +```go +state(initialHeight) = InitialState +state(h+1) <- Execute(state(h), ABCIApp, block(h)) +``` + +where `InitialState` includes the initial consensus parameters and validator set, +and `ABCIApp` is an ABCI application that can return results and changes to the validator +set (TODO). Execute is defined as: + +```go +func Execute(state State, app ABCIApp, block Block) State { + // Fuction ApplyBlock executes block of transactions against the app and returns the new root hash of the app state, + // modifications to the validator set and the changes of the consensus parameters. + AppHash, ValidatorChanges, ConsensusParamChanges := app.ApplyBlock(block) + + nextConsensusParams := UpdateConsensusParams(state.ConsensusParams, ConsensusParamChanges) + return State{ + ChainID: state.ChainID, + InitialHeight: state.InitialHeight, + LastResults: abciResponses.DeliverTxResults, + AppHash: AppHash, + LastValidators: state.Validators, + Validators: state.NextValidators, + NextValidators: UpdateValidators(state.NextValidators, ValidatorChanges), + ConsensusParams: nextConsensusParams, + Version: { + Consensus: { + AppVersion: nextConsensusParams.Version.AppVersion, + }, + }, + } +} +``` + +Validating a new block is first done prior to the `prevote`, `precommit` & `finalizeCommit` stages. + +The steps to validate a new block are: + +- Check the validity rules of the block and its fields. +- Check the versions (Block & App) are the same as in local state. +- Check the chainID's match. +- Check the height is correct. +- Check the `LastBlockID` corresponds to BlockID currently in state. +- Check the hashes in the header match those in state. +- Verify the LastCommit against state, this step is skipped for the initial height. + - This is where checking the signatures correspond to the correct block will be made. +- Make sure the proposer is part of the validator set. +- Validate bock time. + - Make sure the new blocks time is after the previous blocks time. + - Calculate the medianTime and check it against the blocks time. + - If the blocks height is the initial height then check if it matches the genesis time. +- Validate the evidence in the block. Note: Evidence can be empty + +## Header + +A block header contains metadata about the block and about the consensus, as well as commitments to +the data in the current block, the previous block, and the results returned by the application: + +| Name | Type | Description | Validation | +|-------------------|---------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Version | [Version](#version) | Version defines the application and block versions being used. | Must adhere to the validation rules of [Version](#version) | +| ChainID | String | ChainID is the ID of the chain. This must be unique to your chain. | ChainID must be less than 50 bytes. | +| Height | uint64 | Height is the height for this header. | Must be > 0, >= initialHeight, and == previous Height+1 | +| Time | [Time](#time) | The timestamp is equal to the weighted median of validators present in the last commit. Read more on time in the [BFT-time section](../consensus/bft-time.md). Note: the timestamp of a vote must be greater by at least one millisecond than that of the block being voted on. | Time must be >= previous header timestamp + consensus parameters TimeIotaMs. The timestamp of the first block must be equal to the genesis time (since there's no votes to compute the median). | +| LastBlockID | [BlockID](#blockid) | BlockID of the previous block. | Must adhere to the validation rules of [blockID](#blockid). The first block has `block.Header.LastBlockID == BlockID{}`. | +| LastCommitHash | slice of bytes (`[]byte`) | MerkleRoot of the lastCommit's signatures. The signatures represent the validators that committed to the last block. The first block has an empty slices of bytes for the hash. | Must be of length 32 | +| DataHash | slice of bytes (`[]byte`) | MerkleRoot of the hash of transactions. **Note**: The transactions are hashed before being included in the merkle tree, the leaves of the Merkle tree are the hashes, not the transactions themselves. | Must be of length 32 | +| ValidatorHash | slice of bytes (`[]byte`) | MerkleRoot of the current validator set. The validators are first sorted by voting power (descending), then by address (ascending) prior to computing the MerkleRoot. | Must be of length 32 | +| NextValidatorHash | slice of bytes (`[]byte`) | MerkleRoot of the next validator set. The validators are first sorted by voting power (descending), then by address (ascending) prior to computing the MerkleRoot. | Must be of length 32 | +| ConsensusHash | slice of bytes (`[]byte`) | Hash of the protobuf encoded consensus parameters. | Must be of length 32 | +| AppHash | slice of bytes (`[]byte`) | Arbitrary byte array returned by the application after executing and commiting the previous block. It serves as the basis for validating any merkle proofs that comes from the ABCI application and represents the state of the actual application rather than the state of the blockchain itself. The first block's `block.Header.AppHash` is given by `ResponseInitChain.app_hash`. | This hash is determined by the application, CometBFT can not perform validation on it. | +| LastResultHash | slice of bytes (`[]byte`) | `LastResultsHash` is the root hash of a Merkle tree built from `ResponseDeliverTx` responses (`Log`,`Info`, `Codespace` and `Events` fields are ignored). | Must be of length 32. The first block has `block.Header.ResultsHash == MerkleRoot(nil)`, i.e. the hash of an empty input, for RFC-6962 conformance. | +| EvidenceHash | slice of bytes (`[]byte`) | MerkleRoot of the evidence of Byzantine behavior included in this block. | Must be of length 32 | +| ProposerAddress | slice of bytes (`[]byte`) | Address of the original proposer of the block. Validator must be in the current validatorSet. | Must be of length 20 | + +## Version + +NOTE: that this is more specifically the consensus version and doesn't include information like the +P2P Version. (TODO: we should write a comprehensive document about +versioning that this can refer to) + +| Name | type | Description | Validation | +|-------|--------|---------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------| +| Block | uint64 | This number represents the block version and must be the same throughout an operational network | Must be equal to block version being used in a network (`block.Version.Block == state.Version.Consensus.Block`) | +| App | uint64 | App version is decided on by the application. Read [here](https://github.com/cometbft/cometbft/blob/v0.38.x/spec/abci/abci++_app_requirements.md) | `block.Version.App == state.Version.Consensus.App` | + +## BlockID + +The `BlockID` contains two distinct Merkle roots of the block. The `BlockID` includes these two hashes, as well as the number of parts (ie. `len(MakeParts(block))`) + +| Name | Type | Description | Validation | +|---------------|---------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------| +| Hash | slice of bytes (`[]byte`) | MerkleRoot of all the fields in the header (ie. `MerkleRoot(header)`. | hash must be of length 32 | +| PartSetHeader | [PartSetHeader](#partsetheader) | Used for secure gossiping of the block during consensus, is the MerkleRoot of the complete serialized block cut into parts (ie. `MerkleRoot(MakeParts(block))`). | Must adhere to the validation rules of [PartSetHeader](#partsetheader) | + +See [MerkleRoot](./encoding.md#merkleroot) for details. + +## PartSetHeader + +| Name | Type | Description | Validation | +|-------|---------------------------|-----------------------------------|----------------------| +| Total | int32 | Total amount of parts for a block | Must be > 0 | +| Hash | slice of bytes (`[]byte`) | MerkleRoot of a serialized block | Must be of length 32 | + +## Part + +Part defines a part of a block. In CometBFT blocks are broken into `parts` for gossip. + +| Name | Type | Description | Validation | +|-------|-----------------|-----------------------------------|----------------------| +| index | int32 | Total amount of parts for a block | Must be > 0 | +| bytes | bytes | MerkleRoot of a serialized block | Must be of length 32 | +| proof | [Proof](#proof) | MerkleRoot of a serialized block | Must be of length 32 | + +## Time + +CometBFT uses the [Google.Protobuf.Timestamp](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#google.protobuf.Timestamp) +format, which uses two integers, one 64 bit integer for Seconds and a 32 bit integer for Nanoseconds. + +## Data + +Data is just a wrapper for a list of transactions, where transactions are arbitrary byte arrays: + +| Name | Type | Description | Validation | +|------|----------------------------|------------------------|-----------------------------------------------------------------------------| +| Txs | Matrix of bytes ([][]byte) | Slice of transactions. | Validation does not occur on this field, this data is unknown to CometBFT | + +## Commit + +Commit is a simple wrapper for a list of signatures, with one for each validator. It also contains the relevant BlockID, height and round: + +| Name | Type | Description | Validation | +|------------|----------------------------------|----------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------| +| Height | int64 | Height at which this commit was created. | Must be > 0 | +| Round | int32 | Round that the commit corresponds to. | Must be > 0 | +| BlockID | [BlockID](#blockid) | The blockID of the corresponding block. | Must adhere to the validation rules of [BlockID](#blockid). | +| Signatures | Array of [CommitSig](#commitsig) | Array of commit signatures that correspond to current validator set. | Length of signatures must be > 0 and adhere to the validation of each individual [Commitsig](#commitsig) | + +## ExtendedCommit + +`ExtendedCommit`, similarly to Commit, wraps a list of votes with signatures together with other data needed to verify them. +In addition, it contains the verified vote extensions, one for each non-`nil` vote, along with the extension signatures. + +| Name | Type | Description | Validation | +|--------------------|------------------------------------------|-------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------| +| Height | int64 | Height at which this commit was created. | Must be > 0 | +| Round | int32 | Round that the commit corresponds to. | Must be > 0 | +| BlockID | [BlockID](#blockid) | The blockID of the corresponding block. | Must adhere to the validation rules of [BlockID](#blockid). | +| ExtendedSignatures | Array of [ExtendedCommitSig](#commitsig) | The current validator set's commit signatures, extension, and extension signatures. | Length of signatures must be > 0 and adhere to the validation of each individual [ExtendedCommitSig](#extendedcommitsig) | + +## CommitSig + +`CommitSig` represents a signature of a validator, who has voted either for nil, +a particular `BlockID` or was absent. It's a part of the `Commit` and can be used +to reconstruct the vote set given the validator set. + +| Name | Type | Description | Validation | +|------------------|-----------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------| +| BlockIDFlag | [BlockIDFlag](#blockidflag) | Represents the validators participation in consensus: its vote was not received, voted for the block that received the majority, or voted for nil | Must be one of the fields in the [BlockIDFlag](#blockidflag) enum | +| ValidatorAddress | [Address](#address) | Address of the validator | Must be of length 20 | +| Timestamp | [Time](#time) | This field will vary from `CommitSig` to `CommitSig`. It represents the timestamp of the validator. | [Time](#time) | +| Signature | [Signature](#signature) | Signature corresponding to the validators participation in consensus. | The length of the signature must be > 0 and < than 64 | + +NOTE: `ValidatorAddress` and `Timestamp` fields may be removed in the future +(see [ADR-25](https://github.com/cometbft/cometbft/blob/v0.38.x/docs/architecture/tendermint-core/adr-025-commit.md)). + +## ExtendedCommitSig + +`ExtendedCommitSig` represents a signature of a validator that has voted either for `nil`, +a particular `BlockID` or was absent. It is part of the `ExtendedCommit` and can be used +to reconstruct the vote set given the validator set. +Additionally it contains the vote extensions that were attached to each non-`nil` precommit vote. +All these extensions have been verified by the application operating at the signing validator's node. + +| Name | Type | Description | Validation | +|--------------------|-----------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------| +| BlockIDFlag | [BlockIDFlag](#blockidflag) | Represents the validators participation in consensus: its vote was not received, voted for the block that received the majority, or voted for nil | Must be one of the fields in the [BlockIDFlag](#blockidflag) enum | +| ValidatorAddress | [Address](#address) | Address of the validator | Must be of length 20 | +| Timestamp | [Time](#time) | This field will vary from `CommitSig` to `CommitSig`. It represents the timestamp of the validator. | | +| Signature | [Signature](#signature) | Signature corresponding to the validators participation in consensus. | Length must be > 0 and < 64 | +| Extension | bytes | Vote extension provided by the Application running on the sender of the precommit vote, and verified by the local application. | Length must be zero if BlockIDFlag is not `Commit` | +| ExtensionSignature | [Signature](#signature) | Signature of the vote extension. | Length must be > 0 and < than 64 if BlockIDFlag is `Commit`, else 0 | + +## BlockIDFlag + +BlockIDFlag represents which BlockID the [signature](#commitsig) is for. + +```go +enum BlockIDFlag { + BLOCK_ID_FLAG_UNKNOWN = 0; // indicates an error condition + BLOCK_ID_FLAG_ABSENT = 1; // the vote was not received + BLOCK_ID_FLAG_COMMIT = 2; // voted for the block that received the majority + BLOCK_ID_FLAG_NIL = 3; // voted for nil +} +``` + +## Vote + +A vote is a signed message from a validator for a particular block. +The vote includes information about the validator signing it. When stored in the blockchain or propagated over the network, votes are encoded in Protobuf. + +| Name | Type | Description | Validation | +|--------------------|---------------------------------|------------------------------------------------------------------------------------------|------------------------------------------| +| Type | [SignedMsgType](#signedmsgtype) | The type of message the vote refers to | Must be `PrevoteType` or `PrecommitType` | +| Height | int64 | Height for which this vote was created for | Must be > 0 | +| Round | int32 | Round that the commit corresponds to. | Must be > 0 | +| BlockID | [BlockID](#blockid) | The blockID of the corresponding block. | | +| Timestamp | [Time](#time) | Timestamp represents the time at which a validator signed. | | +| ValidatorAddress | bytes | Address of the validator | Length must be equal to 20 | +| ValidatorIndex | int32 | Index at a specific block height corresponding to the Index of the validator in the set. | Must be > 0 | +| Signature | bytes | Signature by the validator if they participated in consensus for the associated block. | Length must be > 0 and < 64 | +| Extension | bytes | Vote extension provided by the Application running at the validator's node. | Length can be 0 | +| ExtensionSignature | bytes | Signature for the extension | Length must be > 0 and < 64 | + +## CanonicalVote + +CanonicalVote is for validator signing. This type will not be present in a block. +Votes are represented via `CanonicalVote` and also encoded using protobuf via `type.SignBytes` which includes the `ChainID`, +and uses a different ordering of the fields. + +| Name | Type | Description | Validation | +|-----------|---------------------------------|-----------------------------------------|------------------------------------------| +| Type | [SignedMsgType](#signedmsgtype) | The type of message the vote refers to | Must be `PrevoteType` or `PrecommitType` | +| Height | int64 | Height in which the vote was provided. | Must be > 0 | +| Round | int64 | Round in which the vote was provided. | Must be > 0 | +| BlockID | string | ID of the block the vote refers to. | | +| Timestamp | string | Time of the vote. | | +| ChainID | string | ID of the blockchain running consensus. | | + +For signing, votes are represented via [`CanonicalVote`](#canonicalvote) and also encoded using protobuf via +`type.SignBytes` which includes the `ChainID`, and uses a different ordering of +the fields. + +We define a method `Verify` that returns `true` if the signature verifies against the pubkey for the `SignBytes` +using the given ChainID: + +```go +func (vote *Vote) Verify(chainID string, pubKey crypto.PubKey) error { + if !bytes.Equal(pubKey.Address(), vote.ValidatorAddress) { + return ErrVoteInvalidValidatorAddress + } + v := vote.ToProto() + if !pubKey.VerifyBytes(types.VoteSignBytes(chainID, v), vote.Signature) { + return ErrVoteInvalidSignature + } + return nil +} +``` + +### CanonicalVoteExtension + +Vote extensions are signed using a representation similar to votes. +This is the structure to marshall in order to obtain the bytes to sign or verify the signature. + +| Name | Type | Description | Validation | +|-----------|--------|---------------------------------------------|----------------------| +| Extension | bytes | Vote extension provided by the Application. | Can have zero length | +| Height | int64 | Height in which the extension was provided. | Must be > 0 | +| Round | int64 | Round in which the extension was provided. | Must be > 0 | +| ChainID | string | ID of the blockchain running consensus. | | + +## Proposal + +Proposal contains height and round for which this proposal is made, BlockID as a unique identifier +of proposed block, timestamp, and POLRound (a so-called Proof-of-Lock (POL) round) that is needed for +termination of the consensus. If POLRound >= 0, then BlockID corresponds to the block that +is locked in POLRound. The message is signed by the validator private key. + +| Name | Type | Description | Validation | +|-----------|---------------------------------|---------------------------------------------------------------------------------------|---------------------------------------------------------| +| Type | [SignedMsgType](#signedmsgtype) | Represents a Proposal [SignedMsgType](#signedmsgtype) | Must be `ProposalType` [signedMsgType](#signedmsgtype) | +| Height | uint64 | Height for which this vote was created for | Must be > 0 | +| Round | int32 | Round that the commit corresponds to. | Must be > 0 | +| POLRound | int64 | Proof of lock | Must be > 0 | +| BlockID | [BlockID](#blockid) | The blockID of the corresponding block. | [BlockID](#blockid) | +| Timestamp | [Time](#time) | Timestamp represents the time at which a validator signed. | [Time](#time) | +| Signature | slice of bytes (`[]byte`) | Signature by the validator if they participated in consensus for the associated bock. | Length of signature must be > 0 and < 64 | + +## SignedMsgType + +Signed message type represents a signed messages in consensus. + +```proto +enum SignedMsgType { + + SIGNED_MSG_TYPE_UNKNOWN = 0; + // Votes + SIGNED_MSG_TYPE_PREVOTE = 1; + SIGNED_MSG_TYPE_PRECOMMIT = 2; + + // Proposal + SIGNED_MSG_TYPE_PROPOSAL = 32; +} +``` + +## Signature + +Signatures in CometBFT are raw bytes representing the underlying signature. + +See the [signature spec](./encoding.md#key-types) for more. + +## EvidenceList + +EvidenceList is a simple wrapper for a list of evidence: + +| Name | Type | Description | Validation | +|----------|--------------------------------|----------------------------------------|-----------------------------------------------------------------| +| Evidence | Array of [Evidence](#evidence) | List of verified [evidence](#evidence) | Validation adheres to individual types of [Evidence](#evidence) | + +## Evidence + +Evidence in CometBFT is used to indicate breaches in the consensus by a validator. + +More information on how evidence works in CometBFT can be found [here](../consensus/evidence.md) + +### DuplicateVoteEvidence + +`DuplicateVoteEvidence` represents a validator that has voted for two different blocks +in the same round of the same height. Votes are lexicographically sorted on `BlockID`. + +| Name | Type | Description | Validation | +|------------------|---------------|--------------------------------------------------------------------|-----------------------------------------------------| +| VoteA | [Vote](#vote) | One of the votes submitted by a validator when they equivocated | VoteA must adhere to [Vote](#vote) validation rules | +| VoteB | [Vote](#vote) | The second vote submitted by a validator when they equivocated | VoteB must adhere to [Vote](#vote) validation rules | +| TotalVotingPower | int64 | The total power of the validator set at the height of equivocation | Must be equal to nodes own copy of the data | +| ValidatorPower | int64 | Power of the equivocating validator at the height | Must be equal to the nodes own copy of the data | +| Timestamp | [Time](#time) | Time of the block where the equivocation occurred | Must be equal to the nodes own copy of the data | + +### LightClientAttackEvidence + +`LightClientAttackEvidence` is a generalized evidence that captures all forms of known attacks on +a light client such that a full node can verify, propose and commit the evidence on-chain for +punishment of the malicious validators. There are three forms of attacks: Lunatic, Equivocation +and Amnesia. These attacks are exhaustive. You can find a more detailed overview of this [here](../light-client/accountability#the-misbehavior-of-faulty-validators) + +| Name | Type | Description | Validation | +|----------------------|------------------------------------|----------------------------------------------------------------------|------------------------------------------------------------------| +| ConflictingBlock | [LightBlock](#lightblock) | Read Below | Must adhere to the validation rules of [lightBlock](#lightblock) | +| CommonHeight | int64 | Read Below | must be > 0 | +| Byzantine Validators | Array of [Validators](#validator) | validators that acted maliciously | Read Below | +| TotalVotingPower | int64 | The total power of the validator set at the height of the infraction | Must be equal to the nodes own copy of the data | +| Timestamp | [Time](#time) | Time of the block where the infraction occurred | Must be equal to the nodes own copy of the data | + +## LightBlock + +LightBlock is the core data structure of the [light client](../light-client/README.md). It combines two data structures needed for verification ([signedHeader](#signedheader) & [validatorSet](#validatorset)). + +| Name | Type | Description | Validation | +|--------------|-------------------------------|----------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------| +| SignedHeader | [SignedHeader](#signedheader) | The header and commit, these are used for verification purposes. To find out more visit [light client docs](../light-client/README.md) | Must not be nil and adhere to the validation rules of [signedHeader](#signedheader) | +| ValidatorSet | [ValidatorSet](#validatorset) | The validatorSet is used to help with verify that the validators in that committed the infraction were truly in the validator set. | Must not be nil and adhere to the validation rules of [validatorSet](#validatorset) | + +The `SignedHeader` and `ValidatorSet` are linked by the hash of the validator set(`SignedHeader.ValidatorsHash == ValidatorSet.Hash()`. + +## SignedHeader + +The SignedhHeader is the [header](#header) accompanied by the commit to prove it. + +| Name | Type | Description | Validation | +|--------|-------------------|-------------------|-----------------------------------------------------------------------------------| +| Header | [Header](#header) | [Header](#header) | Header cannot be nil and must adhere to the [Header](#header) validation criteria | +| Commit | [Commit](#commit) | [Commit](#commit) | Commit cannot be nil and must adhere to the [Commit](#commit) criteria | + +## ValidatorSet + +| Name | Type | Description | Validation | +|------------|----------------------------------|----------------------------------------------------|-------------------------------------------------------------------------------------------------------------------| +| Validators | Array of [validator](#validator) | List of the active validators at a specific height | The list of validators can not be empty or nil and must adhere to the validation rules of [validator](#validator) | +| Proposer | [validator](#validator) | The block proposer for the corresponding block | The proposer cannot be nil and must adhere to the validation rules of [validator](#validator) | + +## Validator + +| Name | Type | Description | Validation | +|------------------|---------------------------|---------------------------------------------------------------------------------------------------|---------------------------------------------------| +| Address | [Address](#address) | Validators Address | Length must be of size 20 | +| Pubkey | slice of bytes (`[]byte`) | Validators Public Key | must be a length greater than 0 | +| VotingPower | int64 | Validators voting power | cannot be < 0 | +| ProposerPriority | int64 | Validators proposer priority. This is used to gauge when a validator is up next to propose blocks | No validation, value can be negative and positive | + +## Address + +Address is a type alias of a slice of bytes. The address is calculated by hashing the public key using sha256 and truncating it to only use the first 20 bytes of the slice. + +```go +const ( + TruncatedSize = 20 +) + +func SumTruncated(bz []byte) []byte { + hash := sha256.Sum256(bz) + return hash[:TruncatedSize] +} +``` + +## ConsensusParams + +| Name | Type | Description | Field Number | +|-----------|-------------------------------------|------------------------------------------------------------------------------|--------------| +| block | [BlockParams](#blockparams) | Parameters limiting the size of a block and time between consecutive blocks. | 1 | +| evidence | [EvidenceParams](#evidenceparams) | Parameters limiting the validity of evidence of byzantine behavior. | 2 | +| validator | [ValidatorParams](#validatorparams) | Parameters limiting the types of public keys validators can use. | 3 | +| version | [BlockParams](#blockparams) | The ABCI application version. | 4 | + +### BlockParams + +| Name | Type | Description | Field Number | +|--------------|-------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------| +| max_bytes | int64 | Max size of a block, in bytes. | 1 | +| max_gas | int64 | Max sum of `GasWanted` in a proposed block. NOTE: blocks that violate this may be committed if there are Byzantine proposers. It's the application's responsibility to handle this when processing a block! | 2 | + +### EvidenceParams + +| Name | Type | Description | Field Number | +|--------------------|------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------| +| max_age_num_blocks | int64 | Max age of evidence, in blocks. | 1 | +| max_age_duration | [google.protobuf.Duration](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#google.protobuf.Duration) | Max age of evidence, in time. It should correspond with an app's "unbonding period" or other similar mechanism for handling [Nothing-At-Stake attacks](https://vitalik.ca/general/2017/12/31/pos_faq.html#what-is-the-nothing-at-stake-problem-and-how-can-it-be-fixed). | 2 | +| max_bytes | int64 | maximum size in bytes of total evidence allowed to be entered into a block | 3 | + +### ValidatorParams + +| Name | Type | Description | Field Number | +|---------------|-----------------|-----------------------------------------------------------------------|--------------| +| pub_key_types | repeated string | List of accepted public key types. Uses same naming as `PubKey.Type`. | 1 | + +### VersionParams + +| Name | Type | Description | Field Number | +|-------------|--------|-------------------------------|--------------| +| app_version | uint64 | The ABCI application version. | 1 | + +## Proof + +| Name | Type | Description | Field Number | +|-----------|----------------|-----------------------------------------------|--------------| +| total | int64 | Total number of items. | 1 | +| index | int64 | Index item to prove. | 2 | +| leaf_hash | bytes | Hash of item value. | 3 | +| aunts | repeated bytes | Hashes from leaf's sibling to a root's child. | 4 | diff --git a/cometbft/v0.38/spec/core/Overview.mdx b/cometbft/v0.38/spec/core/Overview.mdx new file mode 100644 index 00000000..37048a2f --- /dev/null +++ b/cometbft/v0.38/spec/core/Overview.mdx @@ -0,0 +1,13 @@ +--- +order: 1 +parent: + title: Core + order: 3 +--- + +This section describes the core types and functionality of the CometBFT protocol implementation. + +- [Core Data Structures](/cometbft/v0.38/spec/core/Data_structures) +- [Encoding](/cometbft/v0.38/spec/core/encoding) +- [Genesis](/cometbft/v0.38/spec/core/genesis) +- [State](/cometbft/v0.38/spec/core/state) \ No newline at end of file diff --git a/cometbft/v0.38/spec/core/encoding.mdx b/cometbft/v0.38/spec/core/encoding.mdx new file mode 100644 index 00000000..beac7171 --- /dev/null +++ b/cometbft/v0.38/spec/core/encoding.mdx @@ -0,0 +1,304 @@ +--- +order: 2 +--- + +# Encoding + +## Protocol Buffers + +CometBFT uses [Protocol Buffers](https://developers.google.com/protocol-buffers), specifically proto3, for all data structures. + +Please see the [Proto3 language guide](https://developers.google.com/protocol-buffers/docs/proto3) for more details. + +## Byte Arrays + +The encoding of a byte array is simply the raw-bytes prefixed with the length of +the array as a `UVarint` (what proto calls a `Varint`). + +For details on varints, see the [protobuf +spec](https://developers.google.com/protocol-buffers/docs/encoding#varints). + +For example, the byte-array `[0xA, 0xB]` would be encoded as `0x020A0B`, +while a byte-array containing 300 entires beginning with `[0xA, 0xB, ...]` would +be encoded as `0xAC020A0B...` where `0xAC02` is the UVarint encoding of 300. + +## Hashing + +CometBFT uses `SHA256` as its hash function. +Objects are always serialized before being hashed. +So `SHA256(obj)` is short for `SHA256(ProtoEncoding(obj))`. + +## Public Key Cryptography + +CometBFT uses Protobuf [Oneof](https://developers.google.com/protocol-buffers/docs/proto3#oneof) +to distinguish between different types public keys, and signatures. +Additionally, for each public key, CometBFT +defines an Address function that can be used as a more compact identifier in +place of the public key. Here we list the concrete types, their names, +and prefix bytes for public keys and signatures, as well as the address schemes +for each PubKey. Note for brevity we don't +include details of the private keys beyond their type and name. + +### Key Types + +Each type specifies it's own pubkey, address, and signature format. + +#### Ed25519 + +The address is the first 20-bytes of the SHA256 hash of the raw 32-byte public key: + +```go +address = SHA256(pubkey)[:20] +``` + +The signature is the raw 64-byte ED25519 signature. + +CometBFT adopts [zip215](https://zips.z.cash/zip-0215) for verification of ed25519 signatures. + +> Note: This change will be released in the next major release of CometBFT. + +#### Secp256k1 + +The address is the first 20-bytes of the SHA256 hash of the raw 32-byte public key: + +```go +address = SHA256(pubkey)[:20] +``` + +## Other Common Types + +### BitArray + +The BitArray is used in some consensus messages to represent votes received from +validators, or parts received in a block. It is represented +with a struct containing the number of bits (`Bits`) and the bit-array itself +encoded in base64 (`Elems`). + +| Name | Type | +|-------|----------------------------| +| bits | int64 | +| elems | slice of int64 (`[]int64`) | + +Note BitArray receives a special JSON encoding in the form of `x` and `_` +representing `1` and `0`. Ie. the BitArray `10110` would be JSON encoded as +`"x_xx_"` + +### Part + +Part is used to break up blocks into pieces that can be gossiped in parallel +and securely verified using a Merkle tree of the parts. + +Part contains the index of the part (`Index`), the actual +underlying data of the part (`Bytes`), and a Merkle proof that the part is contained in +the set (`Proof`). + +| Name | Type | +|-------|---------------------------| +| index | uint32 | +| bytes | slice of bytes (`[]byte`) | +| proof | [proof](#merkle-proof) | + +See details of SimpleProof, below. + +### MakeParts + +Encode an object using Protobuf and slice it into parts. +CometBFT uses a part size of 65536 bytes, and allows a maximum of 1601 parts +(see `types.MaxBlockPartsCount`). This corresponds to the hard-coded block size +limit of 100MB. + +```go +func MakeParts(block Block) []Part +``` + +## Merkle Trees + +For an overview of Merkle trees, see +[wikipedia](https://en.wikipedia.org/wiki/Merkle_tree) + +We use the RFC 6962 specification of a merkle tree, with sha256 as the hash function. +Merkle trees are used throughout CometBFT to compute a cryptographic digest of a data structure. +The differences between RFC 6962 and the simplest form a merkle tree are that: + +1. leaf nodes and inner nodes have different hashes. + This is for "second pre-image resistance", to prevent the proof to an inner node being valid as the proof of a leaf. + The leaf nodes are `SHA256(0x00 || leaf_data)`, and inner nodes are `SHA256(0x01 || left_hash || right_hash)`. + +2. When the number of items isn't a power of two, the left half of the tree is as big as it could be. + (The largest power of two less than the number of items) This allows new leaves to be added with less + recomputation. For example: + +```md + Simple Tree with 6 items Simple Tree with 7 items + + * * + / \ / \ + / \ / \ + / \ / \ + / \ / \ + * * * * + / \ / \ / \ / \ + / \ / \ / \ / \ + / \ / \ / \ / \ + * * h4 h5 * * * h6 + / \ / \ / \ / \ / \ +h0 h1 h2 h3 h0 h1 h2 h3 h4 h5 +``` + +### MerkleRoot + +The function `MerkleRoot` is a simple recursive function defined as follows: + +```go +// SHA256([]byte{}) +func emptyHash() []byte { + return tmhash.Sum([]byte{}) +} + +// SHA256(0x00 || leaf) +func leafHash(leaf []byte) []byte { + return tmhash.Sum(append(0x00, leaf...)) +} + +// SHA256(0x01 || left || right) +func innerHash(left []byte, right []byte) []byte { + return tmhash.Sum(append(0x01, append(left, right...)...)) +} + +// largest power of 2 less than k +func getSplitPoint(k int) { ... } + +func MerkleRoot(items [][]byte) []byte{ + switch len(items) { + case 0: + return empthHash() + case 1: + return leafHash(items[0]) + default: + k := getSplitPoint(len(items)) + left := MerkleRoot(items[:k]) + right := MerkleRoot(items[k:]) + return innerHash(left, right) + } +} +``` + +Note: `MerkleRoot` operates on items which are arbitrary byte arrays, not +necessarily hashes. For items which need to be hashed first, we introduce the +`Hashes` function: + +```go +func Hashes(items [][]byte) [][]byte { + return SHA256 of each item +} +``` + +Note: we will abuse notion and invoke `MerkleRoot` with arguments of type `struct` or type `[]struct`. +For `struct` arguments, we compute a `[][]byte` containing the protobuf encoding of each +field in the struct, in the same order the fields appear in the struct. +For `[]struct` arguments, we compute a `[][]byte` by protobuf encoding the individual `struct` elements. + +### Merkle Proof + +Proof that a leaf is in a Merkle tree is composed as follows: + +| Name | Type | +|----------|----------------------------| +| total | int64 | +| index | int64 | +| leafHash | slice of bytes (`[]byte`) | +| aunts | Matrix of bytes ([][]byte) | + +Which is verified as follows: + +```golang +func (proof Proof) Verify(rootHash []byte, leaf []byte) bool { + assert(proof.LeafHash, leafHash(leaf) + + computedHash := computeHashFromAunts(proof.Index, proof.Total, proof.LeafHash, proof.Aunts) + return computedHash == rootHash +} + +func computeHashFromAunts(index, total int, leafHash []byte, innerHashes [][]byte) []byte{ + assert(index < total && index >= 0 && total > 0) + + if total == 1{ + assert(len(proof.Aunts) == 0) + return leafHash + } + + assert(len(innerHashes) > 0) + + numLeft := getSplitPoint(total) // largest power of 2 less than total + if index < numLeft { + leftHash := computeHashFromAunts(index, numLeft, leafHash, innerHashes[:len(innerHashes)-1]) + assert(leftHash != nil) + return innerHash(leftHash, innerHashes[len(innerHashes)-1]) + } + rightHash := computeHashFromAunts(index-numLeft, total-numLeft, leafHash, innerHashes[:len(innerHashes)-1]) + assert(rightHash != nil) + return innerHash(innerHashes[len(innerHashes)-1], rightHash) +} +``` + +The number of aunts is limited to 100 (`MaxAunts`) to protect the node against DOS attacks. +This limits the tree size to 2^100 leaves, which should be sufficient for any +conceivable purpose. + +### IAVL+ Tree + +Because CometBFT only uses a Simple Merkle Tree, application developers are expect to use their own Merkle tree in their applications. For example, the IAVL+ Tree - an immutable self-balancing binary tree for persisting application state is used by the [Cosmos SDK](https://github.com/cosmos/cosmos-sdk/blob/ae77f0080a724b159233bd9b289b2e91c0de21b5/docs/interfaces/lite/specification.md) + +## JSON + +CometBFT has its own JSON encoding in order to keep backwards compatibility with the previous RPC layer. + +Registered types are encoded as: + +```json +{ + "type": "", + "value": +} +``` + +For instance, an ED25519 PubKey would look like: + +```json +{ + "type": "tendermint/PubKeyEd25519", + "value": "uZ4h63OFWuQ36ZZ4Bd6NF+/w9fWUwrOncrQsackrsTk=" +} +``` + +Where the `"value"` is the base64 encoding of the raw pubkey bytes, and the +`"type"` is the type name for Ed25519 pubkeys. + +### Signed Messages + +Signed messages (eg. votes, proposals) in the consensus are encoded using protobuf. + +When signing, the elements of a message are re-ordered so the fixed-length fields +are first, making it easy to quickly check the type, height, and round. +The `ChainID` is also appended to the end. +We call this encoding the SignBytes. For instance, SignBytes for a vote is the protobuf encoding of the following struct: + +```protobuf +message CanonicalVote { + SignedMsgType type = 1; + sfixed64 height = 2; // canonicalization requires fixed size encoding here + sfixed64 round = 3; // canonicalization requires fixed size encoding here + CanonicalBlockID block_id = 4; + google.protobuf.Timestamp timestamp = 5; + string chain_id = 6; +} +``` + +The field ordering and the fixed sized encoding for the first three fields is optimized to ease parsing of SignBytes +in HSMs. It creates fixed offsets for relevant fields that need to be read in this context. + +> Note: All canonical messages are length prefixed. + +For more details, see the [signing spec](/cometbft/v0.38/spec/consensus/Validator-Signing). +Also, see the motivating discussion in +[#1622](https://github.com/tendermint/tendermint/issues/1622). diff --git a/cometbft/v0.38/spec/core/genesis.mdx b/cometbft/v0.38/spec/core/genesis.mdx new file mode 100644 index 00000000..843c4f69 --- /dev/null +++ b/cometbft/v0.38/spec/core/genesis.mdx @@ -0,0 +1,38 @@ +--- +order: 3 +--- + +# Genesis + +The genesis file is the starting point of a chain. An application will populate the `app_state` field in the genesis with their required fields. CometBFT is not able to validate this section because it is unaware what application state consists of. + +## Genesis Fields + +- `genesis_time`: The genesis time is the time the blockchain started or will start. If nodes are started before this time they will sit idle until the time specified. +- `chain_id`: The chainid is the chain identifier. Every chain should have a unique identifier. When conducting a fork based upgrade, we recommend changing the chainid to avoid network or consensus errors. +- `initial_height`: This field is the starting height of the blockchain. When conducting a chain restart to avoid restarting at height 1, the network is able to start at a specified height. +- `consensus_params` + - `block` + - `max_bytes`: The max amount of bytes a block can be. + - `max_gas`: The maximum amount of gas that a block can have. + - `time_iota_ms`: This parameter has no value anymore in CometBFT. + +- `evidence` + - `max_age_num_blocks`: After this preset amount of blocks has passed a single piece of evidence is considered invalid + - `max_age_duration`: After this preset amount of time has passed a single piece of evidence is considered invalid. + - `max_bytes`: The max amount of bytes of all evidence included in a block. + +> Note: For evidence to be considered invalid, evidence must be older than both `max_age_num_blocks` and `max_age_duration` + +- `validator` + - `pub_key_types`: Defines which curves are to be accepted as a valid validator consensus key. CometBFT supports ed25519, sr25519 and secp256k1. + +- `version` + - `app_version`: The version of the application. This is set by the application and is used to identify which version of the app a user should be using in order to operate a node. + +- `validators` + - This is an array of validators. This validator set is used as the starting validator set of the chain. This field can be empty, if the application sets the validator set in `InitChain`. + +- `app_hash`: The applications state root hash. This field does not need to be populated at the start of the chain, the application may provide the needed information via `Initchain`. + +- `app_state`: This section is filled in by the application and is unknown to CometBFT. diff --git a/cometbft/v0.38/spec/core/state.mdx b/cometbft/v0.38/spec/core/state.mdx new file mode 100644 index 00000000..9e350a78 --- /dev/null +++ b/cometbft/v0.38/spec/core/state.mdx @@ -0,0 +1,131 @@ +--- +order: 4 +--- + +# State + +The state contains information whose cryptographic digest is included in block headers, and thus is +necessary for validating new blocks. For instance, the validators set and the results of +transactions are never included in blocks, but their Merkle roots are: +the state keeps track of them. + +The `State` object itself is an implementation detail, since it is never +included in a block or gossiped over the network, and we never compute +its hash. The persistence or query interface of the `State` object +is an implementation detail and not included in the specification. +However, the types in the `State` object are part of the specification, since +the Merkle roots of the `State` objects are included in blocks and values are used during +validation. + +```go +type State struct { + ChainID string + InitialHeight int64 + + LastBlockHeight int64 + LastBlockID types.BlockID + LastBlockTime time.Time + + Version Version + LastResults []Result + AppHash []byte + + LastValidators ValidatorSet + Validators ValidatorSet + NextValidators ValidatorSet + + ConsensusParams ConsensusParams +} +``` + +The chain ID and initial height are taken from the genesis file, and not changed again. The +initial height will be `1` in the typical case, `0` is an invalid value. + +Note there is a hard-coded limit of 10000 validators. This is inherited from the +limit on the number of votes in a commit. + +Further information on [`Validator`'s](/cometbft/v0.38/spec/core/Data_structures#validator), +[`ValidatorSet`'s](/cometbft/v0.38/spec/core/Data_structures#validatorset) and +[`ConsensusParams`'s](/cometbft/v0.38/spec/core/Data_structures#consensusparams) can +be found in [data structures](/cometbft/v0.38/spec/core/Data_structures) + +## Execution + +State gets updated at the end of executing a block. Of specific interest is `ResponseEndBlock` and +`ResponseCommit` + +```go +type ResponseEndBlock struct { + ValidatorUpdates []ValidatorUpdate `protobuf:"bytes,1,rep,name=validator_updates,json=validatorUpdates,proto3" json:"validator_updates"` + ConsensusParamUpdates *types1.ConsensusParams `protobuf:"bytes,2,opt,name=consensus_param_updates,json=consensusParamUpdates,proto3" json:"consensus_param_updates,omitempty"` + Events []Event `protobuf:"bytes,3,rep,name=events,proto3" json:"events,omitempty"` +} +``` + +where + +```go +type ValidatorUpdate struct { + PubKey crypto.PublicKey `protobuf:"bytes,1,opt,name=pub_key,json=pubKey,proto3" json:"pub_key"` + Power int64 `protobuf:"varint,2,opt,name=power,proto3" json:"power,omitempty"` +} +``` + +and + +```go +type ResponseCommit struct { + // reserve 1 + Data []byte `protobuf:"bytes,2,opt,name=data,proto3" json:"data,omitempty"` + RetainHeight int64 `protobuf:"varint,3,opt,name=retain_height,json=retainHeight,proto3" json:"retain_height,omitempty"` +} +``` + +`ValidatorUpdates` are used to add and remove validators to the current set as well as update +validator power. Setting validator power to 0 in `ValidatorUpdate` will cause the validator to be +removed. `ConsensusParams` are safely copied across (i.e. if a field is nil it gets ignored) and the +`Data` from the `ResponseCommit` is used as the `AppHash` + +## Version + +```go +type Version struct { + consensus Consensus + software string +} +``` + +[`Consensus`](/cometbft/v0.38/spec/core/Data_structures#version) contains the protocol version for the blockchain and the +application. + +## Block + +The total size of a block is limited in bytes by the `ConsensusParams.Block.MaxBytes`. +Proposed blocks must be less than this size, and will be considered invalid +otherwise. + +The Application may set `ConsensusParams.Block.MaxBytes` to -1. +In that case, the actual block limit is set to 100 MB, +and CometBFT will provide all transactions in the mempool as part of `PrepareProposal`. +The application has to be careful to return a list of transactions in `ResponsePrepareProposal` +whose size is less than or equal to `RequestPrepareProposal.MaxTxBytes`. + +Blocks should additionally be limited by the amount of "gas" consumed by the +transactions in the block, though this is not yet implemented. + +## Evidence + +For evidence in a block to be valid, it must satisfy: + +```go +block.Header.Time-evidence.Time < ConsensusParams.Evidence.MaxAgeDuration && + block.Header.Height-evidence.Height < ConsensusParams.Evidence.MaxAgeNumBlocks +``` + +A block must not contain more than `ConsensusParams.Evidence.MaxBytes` of evidence. This is +implemented to mitigate spam attacks. + +## Validator + +Validators from genesis file and `ResponseEndBlock` must have pubkeys of type ∈ +`ConsensusParams.Validator.PubKeyTypes`. diff --git a/cometbft/v0.38/spec/ivy-proofs/Dockerfile b/cometbft/v0.38/spec/ivy-proofs/Dockerfile new file mode 100644 index 00000000..be60151f --- /dev/null +++ b/cometbft/v0.38/spec/ivy-proofs/Dockerfile @@ -0,0 +1,37 @@ +# we need python2 support, which was dropped after buster: +FROM debian:buster + +RUN echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections +RUN apt-get update +RUN apt-get install -y apt-utils + +# Install and configure locale `en_US.UTF-8` +RUN apt-get install -y locales && \ + sed -i -e "s/# $en_US.*/en_US.UTF-8 UTF-8/" /etc/locale.gen && \ + dpkg-reconfigure --frontend=noninteractive locales && \ + update-locale LANG=en_US.UTF-8 +ENV LANG=en_US.UTF-8 + +RUN apt-get update +RUN apt-get install -y git python2 python-pip g++ cmake python-ply python-tk tix pkg-config libssl-dev python-setuptools + +# create a user: +RUN useradd -ms /bin/bash user +USER user +WORKDIR /home/user + +RUN git clone --recurse-submodules https://github.com/kenmcmil/ivy.git +WORKDIR /home/user/ivy/ +RUN git checkout 271ee38980699115508eb90a0dd01deeb750a94b + +RUN python2.7 build_submodules.py +RUN mkdir -p "/home/user/python/lib/python2.7/site-packages" +ENV PYTHONPATH="/home/user/python/lib/python2.7/site-packages" +# need to install pyparsing manually because otherwise wrong version found +RUN pip install pyparsing +RUN python2.7 setup.py install --prefix="/home/user/python/" +ENV PATH=$PATH:"/home/user/python/bin/" +WORKDIR /home/user/tendermint-proof/ + +ENTRYPOINT ["/home/user/tendermint-proof/check_proofs.sh"] + diff --git a/cometbft/v0.38/spec/ivy-proofs/README.md b/cometbft/v0.38/spec/ivy-proofs/README.md new file mode 100644 index 00000000..00a4bed2 --- /dev/null +++ b/cometbft/v0.38/spec/ivy-proofs/README.md @@ -0,0 +1,33 @@ +# Ivy Proofs + +```copyright +Copyright (c) 2020 Galois, Inc. +SPDX-License-Identifier: Apache-2.0 +``` + +## Contents + +This folder contains: + +* `tendermint.ivy`, a specification of Tendermint algorithm as described in *The latest gossip on BFT consensus* by E. Buchman, J. Kwon, Z. Milosevic. +* `abstract_tendermint.ivy`, a more abstract specification of Tendermint that is more verification-friendly. +* `classic_safety.ivy`, a proof that Tendermint satisfies the classic safety property of BFT consensus: if every two quorums have a well-behaved node in common, then no two well-behaved nodes ever disagree. +* `accountable_safety_1.ivy`, a proof that, assuming every quorum contains at least one well-behaved node, if two well-behaved nodes disagree, then there is evidence demonstrating at least f+1 nodes misbehaved. +* `accountable_safety_2.ivy`, a proof that, regardless of any assumption about quorums, well-behaved nodes cannot be framed by malicious nodes. In other words, malicious nodes can never construct evidence that incriminates a well-behaved node. +* `network_shim.ivy`, the network model and a convenience `shim` object to interface with the Tendermint specification. +* `domain_model.ivy`, a specification of the domain model underlying the Tendermint specification, i.e. rounds, value, quorums, etc. + +All specifications and proofs are written in [Ivy](https://github.com/kenmcmil/ivy). + +The license above applies to all files in this folder. + + +## Building and running + +The easiest way to check the proofs is to use [Docker](https://www.docker.com/). + +1. Install [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/install/). +2. Build a Docker image: `docker-compose build` +3. Run the proofs inside the Docker container: `docker-compose run +tendermint-proof`. This will check all the proofs with the `ivy_check` +command and write the output of `ivy_check` to a subdirectory of `./output/' diff --git a/cometbft/v0.38/spec/ivy-proofs/abstract_tendermint.ivy b/cometbft/v0.38/spec/ivy-proofs/abstract_tendermint.ivy new file mode 100644 index 00000000..4a160be2 --- /dev/null +++ b/cometbft/v0.38/spec/ivy-proofs/abstract_tendermint.ivy @@ -0,0 +1,178 @@ +#lang ivy1.7 +# --- +# layout: page +# title: Abstract specification of Tendermint in Ivy +# --- + +# Here we define an abstract version of the Tendermint specification. We use +# two main forms of abstraction: a) We abstract over how information is +# transmitted (there is no network). b) We abstract functions using relations. +# For example, we abstract over a node's current round, instead only tracking +# with a relation which rounds the node has left. We do something similar for +# the `lockedRound` variable. This is in order to avoid using a function from +# node to round, and it allows us to emit verification conditions that are +# efficiently solvable by Z3. + +# This specification also defines the observations that are used to adjudicate +# misbehavior. Well-behaved nodes faithfully observe every message that they +# use to take a step, while Byzantine nodes can fake observations about +# themselves (including withholding observations). Misbehavior is defined using +# the collection of all observations made (in reality, those observations must +# be collected first, but we do not model this process). + +include domain_model + +module abstract_tendermint = { + +# Protocol state +# ############## + + relation left_round(N:node, R:round) + relation prevoted(N:node, R:round, V:value) + relation precommitted(N:node, R:round, V:value) + relation decided(N:node, R:round, V:value) + relation locked(N:node, R:round, V:value) + +# Accountability relations +# ######################## + + relation observed_prevoted(N:node, R:round, V:value) + relation observed_precommitted(N:node, R:round, V:value) + +# relations that are defined in terms of the previous two: + relation observed_equivocation(N:node) + relation observed_unlawful_prevote(N:node) + relation agreement + relation accountability_violation + + object defs = { # we hide those definitions and use them only when needed + private { + definition [observed_equivocation_def] observed_equivocation(N) = exists V1,V2,R . + V1 ~= V2 & (observed_precommitted(N,R,V1) & observed_precommitted(N,R,V2) | observed_prevoted(N,R,V1) & observed_prevoted(N,R,V2)) + + definition [observed_unlawful_prevote_def] observed_unlawful_prevote(N) = exists V1,V2,R1,R2 . + V1 ~= value.nil & V2 ~= value.nil & V1 ~= V2 & R1 < R2 & observed_precommitted(N,R1,V1) & observed_prevoted(N,R2,V2) + & forall Q,R . R1 <= R & R < R2 & nset.is_quorum(Q) -> exists N2 . nset.member(N2,Q) & ~observed_prevoted(N2,R,V2) + + definition [agreement_def] agreement = forall N1,N2,R1,R2,V1,V2 . well_behaved(N1) & well_behaved(N2) & decided(N1,R1,V1) & decided(N2,R2,V2) -> V1 = V2 + + definition [accountability_violation_def] accountability_violation = exists Q1,Q2 . nset.is_quorum(Q1) & nset.is_quorum(Q2) & (forall N . nset.member(N,Q1) & nset.member(N,Q2) -> observed_equivocation(N) | observed_unlawful_prevote(N)) + } + } + +# Protocol transitions +# #################### + + after init { + left_round(N,R) := R < 0; + prevoted(N,R,V) := false; + precommitted(N,R,V) := false; + decided(N,R,V) := false; + locked(N,R,V) := false; + + observed_prevoted(N,R,V) := false; + observed_precommitted(N,R,V) := false; + } + +# Actions are named after the corresponding line numbers in the Tendermint +# arXiv paper. + + action l_11(n:node, r:round) = { # start round r + require ~left_round(n,r); + left_round(n,R) := R < r; + } + + action l_22(n:node, rp:round, v:value) = { + require ~left_round(n,rp); + require ~prevoted(n,rp,V) & ~precommitted(n,rp,V); + require (forall R,V . locked(n,R,V) -> V = v) | v = value.nil; + prevoted(n, rp, v) := true; + left_round(n, R) := R < rp; # leave all lower rounds. + + observed_prevoted(n, rp, v) := observed_prevoted(n, rp, v) | well_behaved(n); # the node observes itself + } + + action l_28(n:node, rp:round, v:value, vr:round, q:nset) = { + require ~left_round(n,rp) & ~prevoted(n,rp,V); + require ~prevoted(n,rp,V) & ~precommitted(n,rp,V); + require vr < rp; + require nset.is_quorum(q) & (forall N . nset.member(N,q) -> (prevoted(N,vr,v) | ~well_behaved(N))); + var proposal:value; + if value.valid(v) & ((forall R0,V0 . locked(n,R0,V0) -> R0 <= vr) | (forall R,V . locked(n,R,V) -> V = v)) { + proposal := v; + } + else { + proposal := value.nil; + }; + prevoted(n, rp, proposal) := true; + left_round(n, R) := R < rp; # leave all lower rounds + + observed_prevoted(N, vr, v) := observed_prevoted(N, vr, v) | (well_behaved(n) & nset.member(N,q)); # the node observes the prevotes of quorum q + observed_prevoted(n, rp, proposal) := observed_prevoted(n, rp, proposal) | well_behaved(n); # the node observes itself + } + + action l_36(n:node, rp:round, v:value, q:nset) = { + require v ~= value.nil; + require ~left_round(n,rp); + require exists V . prevoted(n,rp,V); + require ~precommitted(n,rp,V); + require nset.is_quorum(q) & (forall N . nset.member(N,q) -> (prevoted(N,rp,v) | ~well_behaved(N))); + precommitted(n, rp, v) := true; + left_round(n, R) := R < rp; # leave all lower rounds + locked(n,R,V) := R <= rp & V = v; + + observed_prevoted(N, rp, v) := observed_prevoted(N, rp, v) | (well_behaved(n) & nset.member(N,q)); # the node observes the prevotes of quorum q + observed_precommitted(n, rp, v) := observed_precommitted(n, rp, v) | well_behaved(n); # the node observes itself + } + + action l_44(n:node, rp:round, q:nset) = { + require ~left_round(n,rp); + require ~precommitted(n,rp,V); + require nset.is_quorum(q) & (forall N .nset.member(N,q) -> (prevoted(N,rp,value.nil) | ~well_behaved(N))); + precommitted(n, rp, value.nil) := true; + left_round(n, R) := R < rp; # leave all lower rounds + + observed_prevoted(N, rp, value.nil) := observed_prevoted(N, rp, value.nil) | (well_behaved(n) & nset.member(N,q)); # the node observes the prevotes of quorum q + observed_precommitted(n, rp, value.nil) := observed_precommitted(n, rp, value.nil) | well_behaved(n); # the node observes itself + } + + action l_57(n:node, rp:round) = { + require ~left_round(n,rp); + require ~prevoted(n,rp,V); + prevoted(n, rp, value.nil) := true; + left_round(n, R) := R < rp; # leave all lower rounds + + observed_prevoted(n, rp, value.nil) := observed_prevoted(n, rp, value.nil) | well_behaved(n); # the node observes itself + } + + action l_61(n:node, rp:round) = { + require ~left_round(n,rp); + require ~precommitted(n,rp,V); + precommitted(n, rp, value.nil) := true; + left_round(n, R) := R < rp; # leave all lower rounds + + observed_precommitted(n, rp, value.nil) := observed_precommitted(n, rp, value.nil) | well_behaved(n); # the node observes itself + } + + action decide(n:node, r:round, v:value, q:nset) = { + require v ~= value.nil; + require nset.is_quorum(q) & (forall N . nset.member(N, q) -> (precommitted(N, r, v) | ~well_behaved(N))); + decided(n, r, v) := true; + + observed_precommitted(N, r, v) := observed_precommitted(N, r, v) | (well_behaved(n) & nset.member(N,q)); # the node observes the precommits of quorum q + + } + + action misbehave = { +# Byzantine nodes can claim they observed whatever they want about themselves, +# but they cannot remove observations. Note that we use assume because we don't +# want those to be checked; we just want them to be true (that's the model of +# Byzantine behavior). + observed_prevoted(N,R,V) := *; + assume (old observed_prevoted(N,R,V)) -> observed_prevoted(N,R,V); + assume well_behaved(N) -> old observed_prevoted(N,R,V) = observed_prevoted(N,R,V); + observed_precommitted(N,R,V) := *; + assume (old observed_precommitted(N,R,V)) -> observed_precommitted(N,R,V); + assume well_behaved(N) -> old observed_precommitted(N,R,V) = observed_precommitted(N,R,V); + } +} diff --git a/cometbft/v0.38/spec/ivy-proofs/accountable_safety_1.ivy b/cometbft/v0.38/spec/ivy-proofs/accountable_safety_1.ivy new file mode 100644 index 00000000..02bdf1ad --- /dev/null +++ b/cometbft/v0.38/spec/ivy-proofs/accountable_safety_1.ivy @@ -0,0 +1,143 @@ +#lang ivy1.7 +# --- +# layout: page +# title: Proof of Classic Safety +# --- + +include tendermint +include abstract_tendermint + +# Here we prove the first accountability property: if two well-behaved nodes +# disagree, then there are two quorums Q1 and Q2 such that all members of the +# intersection of Q1 and Q2 have violated the accountability properties. + +# The proof is done in two steps: first we prove the abstract specification +# satisfies the property, and then we show by refinement that this property +# also holds in the concrete specification. + +# To see what is checked in the refinement proof, use `ivy_show isolate=accountable_safety_1 accountable_safety_1.ivy` +# To see what is checked in the abstract correctness proof, use `ivy_show isolate=abstract_accountable_safety_1 accountable_safety_1.ivy` +# To check the whole proof, use `ivy_check accountable_safety_1.ivy`. + + +# Proof of the accountability property in the abstract specification +# ================================================================== + +# We prove with tactics (see `lemma_1` and `lemma_2`) that, if some basic +# invariants hold (see `invs` below), then the accountability property holds. + +isolate abstract_accountable_safety = { + + instantiate abstract_tendermint + +# The main property +# ----------------- + +# If there is disagreement, then there is evidence that a third of the nodes +# have violated the protocol: + invariant [accountability] agreement | accountability_violation + proof { + apply lemma_1.thm # this reduces to goal to three subgoals: p1, p2, and p3 (see their definition below) + proof [p1] { + assume invs.inv1 + } + proof [p2] { + assume invs.inv2 + } + proof [p3] { + assume invs.inv3 + } + } + +# The invariants +# -------------- + + isolate invs = { + + # well-behaved nodes observe their own actions faithfully: + invariant [inv1] well_behaved(N) -> (observed_precommitted(N,R,V) = precommitted(N,R,V)) + # if a value is precommitted by a well-behaved node, then a quorum is observed to prevote it: + invariant [inv2] (exists N . well_behaved(N) & precommitted(N,R,V)) & V ~= value.nil -> exists Q . nset.is_quorum(Q) & forall N2 . nset.member(N2,Q) -> observed_prevoted(N2,R,V) + # if a value is decided by a well-behaved node, then a quorum is observed to precommit it: + invariant [inv3] (exists N . well_behaved(N) & decided(N,R,V)) -> 0 <= R & V ~= value.nil & exists Q . nset.is_quorum(Q) & forall N2 . nset.member(N2,Q) -> observed_precommitted(N2,R,V) + private { + invariant (precommitted(N,R,V) | prevoted(N,R,V)) -> 0 <= R + invariant R < 0 -> left_round(N,R) + } + + } with this, nset, round, accountable_bft.max_2f_byzantine + +# The theorems proved with tactics +# -------------------------------- + +# Using complete induction on rounds, we prove that, assuming that the +# invariants inv1, inv2, and inv3 hold, the accountability property holds. + +# For technical reasons, we separate the proof in two steps + isolate lemma_1 = { + + specification { + theorem [thm] { + property [p1] forall N,R,V . well_behaved(N) -> (observed_precommitted(N,R,V) = precommitted(N,R,V)) + property [p2] forall R,V . (exists N . well_behaved(N) & precommitted(N,R,V)) & V ~= value.nil -> exists Q . nset.is_quorum(Q) & forall N2 . nset.member(N2,Q) -> observed_prevoted(N2,R,V) + property [p3] forall R,V. (exists N . well_behaved(N) & decided(N,R,V)) -> 0 <= R & V ~= value.nil & exists Q . nset.is_quorum(Q) & forall N2 . nset.member(N2,Q) -> observed_precommitted(N2,R,V) + #------------------------------------------------------------------------------------------------------------------------------------------- + property agreement | accountability_violation + } + proof { + assume inductive_property # the theorem follows from what we prove by induction below + } + } + + implementation { + # complete induction is not built-in, so we introduce it with an axiom. Note that this only holds for a type where 0 is the smallest element + axiom [complete_induction] { + relation p(X:round) + { # base case + property p(0) + } + { # inductive step: show that if the property is true for all X lower or equal to x and y=x+1, then the property is true of y + individual a:round + individual b:round + property (forall X. 0 <= X & X <= a -> p(X)) & round.succ(a,b) -> p(b) + } + #-------------------------- + property forall X . 0 <= X -> p(X) + } + + # The main lemma: if inv1 and inv2 below hold and a quorum is observed to + # precommit V1 at R1 and another quorum is observed to precommit V2~=V1 at + # R2>=R1, then the intersection of two quorums (i.e. f+1 nodes) is observed to + # violate the protocol. We prove this by complete induction on R2. + theorem [inductive_property] { + property [p1] forall N,R,V . well_behaved(N) -> (observed_precommitted(N,R,V) = precommitted(N,R,V)) + property [p2] forall R,V . (exists N . well_behaved(N) & precommitted(N,R,V)) -> V = value.nil | exists Q . nset.is_quorum(Q) & forall N2 . nset.member(N2,Q) -> observed_prevoted(N2,R,V) + #----------------------------------------------------------------------------------------------------------------------- + property forall R2. 0 <= R2 -> ((exists V2,Q1,R1,V1,Q1 . V1 ~= value.nil & V2 ~= value.nil & V1 ~= V2 & 0 <= R1 & R1 <= R2 & nset.is_quorum(Q1) & (forall N . nset.member(N,Q1) -> observed_precommitted(N,R1,V1)) & (exists Q2 . nset.is_quorum(Q2) & forall N . nset.member(N,Q2) -> observed_prevoted(N,R2,V2))) -> accountability_violation) + } + proof { + apply complete_induction # the two subgoals (base case and inductive case) are then discharged automatically + # NOTE: this can take a long time depending on the SMT random seed (to try a different seed, use `ivy_check seed=$RANDOM` + } + } + } with this, round, nset, accountable_bft.max_2f_byzantine, defs.observed_equivocation_def, defs.observed_unlawful_prevote_def, defs.accountability_violation_def, defs.agreement_def + +} with round + +# The final proof +# =============== + +isolate accountable_safety_1 = { + +# First we instantiate the concrete protocol: + instantiate tendermint(abstract_accountable_safety) + +# We then define what we mean by agreement + relation agreement + definition [agreement_def] agreement = forall N1,N2. well_behaved(N1) & well_behaved(N2) & server.decision(N1) ~= value.nil & server.decision(N2) ~= value.nil -> server.decision(N1) = server.decision(N2) + + invariant abstract_accountable_safety.agreement -> agreement + + invariant [accountability] agreement | abstract_accountable_safety.accountability_violation + +} with value, round, proposers, shim, abstract_accountable_safety, abstract_accountable_safety.defs.agreement_def, accountable_safety_1.agreement_def diff --git a/cometbft/v0.38/spec/ivy-proofs/accountable_safety_2.ivy b/cometbft/v0.38/spec/ivy-proofs/accountable_safety_2.ivy new file mode 100644 index 00000000..7fb92890 --- /dev/null +++ b/cometbft/v0.38/spec/ivy-proofs/accountable_safety_2.ivy @@ -0,0 +1,52 @@ +#lang ivy1.7 + +include tendermint +include abstract_tendermint + +# Here we prove the second accountability property: no well-behaved node is +# ever observed to violate the accountability properties. + +# The proof is done in two steps: first we prove the the abstract specification +# satisfies the property, and then we show by refinement that this property +# also holds in the concrete specification. + +# To see what is checked in the refinement proof, use `ivy_show isolate=accountable_safety_2 accountable_safety_2.ivy` +# To see what is checked in the abstract correctness proof, use `ivy_show isolate=abstract_accountable_safety_2 accountable_safety_2.ivy` +# To check the whole proof, use `ivy_check complete=fo accountable_safety_2.ivy`. + +# Proof that the property holds in the abstract specification +# ============================================================ + +isolate abstract_accountable_safety_2 = { + + instantiate abstract_tendermint + +# the main property: + invariant [wb_never_punished] well_behaved(N) -> ~(observed_equivocation(N) | observed_unlawful_prevote(N)) + +# the main invariant for proving wb_not_punished: + invariant well_behaved(N) & precommitted(N,R,V) & ~locked(N,R,V) & V ~= value.nil -> exists R2,V2 . V2 ~= value.nil & R < R2 & precommitted(N,R2,V2) & locked(N,R2,V2) + + invariant (exists N . well_behaved(N) & precommitted(N,R,V) & V ~= value.nil) -> exists Q . nset.is_quorum(Q) & forall N . nset.member(N,Q) -> observed_prevoted(N,R,V) + + invariant well_behaved(N) -> (observed_prevoted(N,R,V) <-> prevoted(N,R,V)) + invariant well_behaved(N) -> (observed_precommitted(N,R,V) <-> precommitted(N,R,V)) + +# nodes stop prevoting or precommitting in lower rounds when doing so in a higher round: + invariant well_behaved(N) & prevoted(N,R2,V2) & R1 < R2 -> left_round(N,R1) + invariant well_behaved(N) & locked(N,R2,V2) & R1 < R2 -> left_round(N,R1) + + invariant [precommit_unique_per_round] well_behaved(N) & precommitted(N,R,V1) & precommitted(N,R,V2) -> V1 = V2 + +} with nset, round, abstract_accountable_safety_2.defs.observed_equivocation_def, abstract_accountable_safety_2.defs.observed_unlawful_prevote_def + +# Proof that the property holds in the concrete specification +# =========================================================== + +isolate accountable_safety_2 = { + + instantiate tendermint(abstract_accountable_safety_2) + + invariant well_behaved(N) -> ~(abstract_accountable_safety_2.observed_equivocation(N) | abstract_accountable_safety_2.observed_unlawful_prevote(N)) + +} with round, value, shim, abstract_accountable_safety_2, abstract_accountable_safety_2.defs.observed_equivocation_def, abstract_accountable_safety_2.defs.observed_unlawful_prevote_def diff --git a/cometbft/v0.38/spec/ivy-proofs/check_proofs.sh b/cometbft/v0.38/spec/ivy-proofs/check_proofs.sh new file mode 100755 index 00000000..6afd1a96 --- /dev/null +++ b/cometbft/v0.38/spec/ivy-proofs/check_proofs.sh @@ -0,0 +1,39 @@ +#!/bin/bash + +# returns non-zero error code if any proof fails + +success=0 +log_dir=$(cat /dev/urandom | tr -cd 'a-f0-9' | head -c 6) +cmd="ivy_check seed=$RANDOM" +mkdir -p output/$log_dir + +echo "Checking classic safety:" +res=$($cmd classic_safety.ivy | tee "output/$log_dir/classic_safety.txt" | tail -n 1) +if [ "$res" = "OK" ]; then + echo "OK" +else + echo "FAILED" + success=1 +fi + +echo "Checking accountable safety 1:" +res=$($cmd accountable_safety_1.ivy | tee "output/$log_dir/accountable_safety_1.txt" | tail -n 1) +if [ "$res" = "OK" ]; then + echo "OK" +else + echo "FAILED" + success=1 +fi + +echo "Checking accountable safety 2:" +res=$($cmd complete=fo accountable_safety_2.ivy | tee "output/$log_dir/accountable_safety_2.txt" | tail -n 1) +if [ "$res" = "OK" ]; then + echo "OK" +else + echo "FAILED" + success=1 +fi + +echo +echo "See ivy_check output in the output/ folder" +exit $success diff --git a/cometbft/v0.38/spec/ivy-proofs/classic_safety.ivy b/cometbft/v0.38/spec/ivy-proofs/classic_safety.ivy new file mode 100644 index 00000000..b422a2c1 --- /dev/null +++ b/cometbft/v0.38/spec/ivy-proofs/classic_safety.ivy @@ -0,0 +1,85 @@ +#lang ivy1.7 +# --- +# layout: page +# title: Proof of Classic Safety +# --- + +include tendermint +include abstract_tendermint + +# Here we prove the classic safety property: assuming that every two quorums +# have a well-behaved node in common, no two well-behaved nodes ever disagree. + +# The proof is done in two steps: first we prove the the abstract specification +# satisfies the property, and then we show by refinement that this property +# also holds in the concrete specification. + +# To see what is checked in the refinement proof, use `ivy_show isolate=classic_safety classic_safety.ivy` +# To see what is checked in the abstract correctness proof, use `ivy_show isolate=abstract_classic_safety classic_safety.ivy` + +# To check the whole proof, use `ivy_check classic_safety.ivy`. + +# Note that all the verification conditions sent to Z3 for this proof are in +# EPR. + +# Classic safety in the abstract model +# ==================================== + +# We start by proving that classic safety holds in the abstract model. + +isolate abstract_classic_safety = { + + instantiate abstract_tendermint + + invariant [classic_safety] classic_bft.quorum_intersection & decided(N1,R1,V1) & decided(N2,R2,V2) -> V1 = V2 + +# The notion of choosable value +# ----------------------------- + + relation choosable(R:round, V:value) + definition choosable(R,V) = exists Q . nset.is_quorum(Q) & forall N . well_behaved(N) & nset.member(N,Q) -> ~left_round(N,R) | precommitted(N,R,V) + +# Main invariants +# --------------- + +# `classic_safety` is inductive relative to those invariants + + invariant [decision_is_quorum_precommit] (exists N1 . decided(N1,R,V)) -> exists Q. nset.is_quorum(Q) & forall N2. well_behaved(N2) & nset.member(N2, Q) -> precommitted(N2,R,V) + + invariant [precommitted_is_quorum_prevote] V ~= value.nil & (exists N1 . precommitted(N1,R,V)) -> exists Q. nset.is_quorum(Q) & forall N2. well_behaved(N2) & nset.member(N2, Q) -> prevoted(N2,R,V) + + invariant [prevote_unique_per_round] prevoted(N,R,V1) & prevoted(N,R,V2) -> V1 = V2 + +# This is the core invariant: as long as a precommitted value is still choosable, it remains protected by a lock and prevents any new value from being prevoted: + invariant [locks] classic_bft.quorum_intersection & V ~= value.nil & precommitted(N,R,V) & choosable(R,V) -> locked(N,R,V) & forall R2,V2 . R < R2 & prevoted(N,R2,V2) -> V2 = V | V2 = value.nil + +# Supporting invariants +# --------------------- + +# The main invariants are inductive relative to those + + invariant decided(N,R,V) -> V ~= value.nil + + invariant left_round(N,R2) & R1 < R2 -> left_round(N,R1) # if a node left round R2>R1, then it also left R1: + + invariant prevoted(N,R2,V2) & R1 < R2 -> left_round(N,R1) + invariant precommitted(N,R2,V2) & R1 < R2 -> left_round(N,R1) + +} with round, nset, classic_bft.quorum_intersection_def + +# The refinement proof +# ==================== + +# Now, thanks to the refinement relation that we establish in +# `concrete_tendermint.ivy`, we prove that classic safety transfers to the +# concrete specification: +isolate classic_safety = { + + # We instantiate the `tendermint` module providing `abstract_classic_safety` as abstract model. + instantiate tendermint(abstract_classic_safety) + + # We prove that if every two quorums have a well-behaved node in common, + # then well-behaved nodes never disagree: + invariant [classic_safety] classic_bft.quorum_intersection & server.decision(N1) ~= value.nil & server.decision(N2) ~= value.nil -> server.decision(N1) = server.decision(N2) + +} with value, round, proposers, shim, abstract_classic_safety # here we list all the specifications that we rely on for this proof diff --git a/cometbft/v0.38/spec/ivy-proofs/count_lines.sh b/cometbft/v0.38/spec/ivy-proofs/count_lines.sh new file mode 100755 index 00000000..b2c457e2 --- /dev/null +++ b/cometbft/v0.38/spec/ivy-proofs/count_lines.sh @@ -0,0 +1,13 @@ +#!/bin/bash + +r='^\s*$\|^\s*\#\|^\s*\}\s*$\|^\s*{\s*$' # removes comments and blank lines and lines that contain only { or } +N1=`cat tendermint.ivy domain_model.ivy network_shim.ivy | grep -v $r'\|.*invariant.*' | wc -l` +N2=`cat abstract_tendermint.ivy | grep "observed_" | wc -l` # the observed_* variables specify the observations of the nodes +SPEC_LINES=`expr $N1 + $N2` +echo "spec lines: $SPEC_LINES" +N3=`cat abstract_tendermint.ivy | grep -v $r'\|.*observed_.*' | wc -l` +N4=`cat accountable_safety_1.ivy | grep -v $r | wc -l` +PROOF_LINES=`expr $N3 + $N4` +echo "proof lines: $PROOF_LINES" +RATIO=`bc <<< "scale=2;$PROOF_LINES / $SPEC_LINES"` +echo "proof-to-code ratio for the accountable-safety property: $RATIO" diff --git a/cometbft/v0.38/spec/ivy-proofs/docker-compose.yml b/cometbft/v0.38/spec/ivy-proofs/docker-compose.yml new file mode 100644 index 00000000..e0612d4b --- /dev/null +++ b/cometbft/v0.38/spec/ivy-proofs/docker-compose.yml @@ -0,0 +1,7 @@ +version: '3' +services: + tendermint-proof: + build: . + volumes: + - ./:/home/user/tendermint-proof:ro + - ./output:/home/user/tendermint-proof/output:rw diff --git a/cometbft/v0.38/spec/ivy-proofs/domain_model.ivy b/cometbft/v0.38/spec/ivy-proofs/domain_model.ivy new file mode 100644 index 00000000..1fd3cc99 --- /dev/null +++ b/cometbft/v0.38/spec/ivy-proofs/domain_model.ivy @@ -0,0 +1,144 @@ +#lang ivy1.7 + +include order # this is a file from the standard library (`ivy/ivy/include/1.7/order.ivy`) + +isolate round = { + type this + individual minus_one:this + relation succ(R1:round, R2:round) + action incr(i:this) returns (j:this) + specification { +# to simplify verification, we treat rounds as an abstract totally ordered set with a successor relation. + instantiate totally_ordered(this) + property minus_one < 0 + property succ(X,Z) -> (X < Z & ~(X < Y & Y < Z)) + after incr { + ensure succ(i,j) + } + } + implementation { +# here we prove that the abstraction is sound. + interpret this -> int # rounds are integers in the Tendermint specification. + definition minus_one = 0-1 + definition succ(R1,R2) = R2 = R1 + 1 + implement incr { + j := i+1; + } + } +} + +instance node : iterable # nodes are a set with an order, that can be iterated over (see order.ivy in the standard library) + +relation well_behaved(N:node) # whether a node is well-behaved or not. NOTE: Used only in the proof and the Byzantine model; Nodes do know know who is well-behaved and who is not. + +isolate proposers = { + # each round has a unique proposer in Tendermint. In order to avoid a + # function from round to node (which makes verification more difficult), we + # abstract over this function using a relation. + relation is_proposer(N:node, R:round) + export action get_proposer(r:round) returns (n:node) + specification { + property is_proposer(N1,R) & is_proposer(N2,R) -> N1 = N2 + after get_proposer { + ensure is_proposer(n,r); + } + } + implementation { + function f(R:round):node + definition f(r:round) = <<>> + definition is_proposer(N,R) = N = f(R) + implement get_proposer { + n := f(r); + } + } +} + +isolate value = { # the type of values + type this + relation valid(V:value) + individual nil:value + specification { + property ~valid(nil) + } + implementation { + interpret value -> bv[2] + definition nil = <<< -1 >>> # let's say nil is -1 + definition valid(V) = V ~= nil + } +} + +object nset = { # the type of node sets + type this # a set of N=3f+i nodes for 0 + #include + namespace hash_space { + template + class hash > { + public: + size_t operator()(const std::set &s) const { + hash h; + size_t res = 0; + for (const T &e : s) + res += h(e); + return res; + } + }; + } + >>> + interpret nset -> <<< std::set<`node`> >>> + definition member(n:node, s:nset) = <<< `s`.find(`n`) != `s`.end() >>> + definition is_quorum(s:nset) = <<< 3*`s`.size() > 2*`node.size` >>> + definition is_blocking(s:nset) = <<< 3*`s`.size() > `node.size` >>> + implement empty { + <<< + >>> + } + implement insert { + <<< + `t` = `s`; + `t`.insert(`n`); + >>> + } + <<< encode `nset` + + std::ostream &operator <<(std::ostream &s, const `nset` &a) { + s << "{"; + for (auto iter = a.begin(); iter != a.end(); iter++) { + if (iter != a.begin()) s << ", "; + s << *iter; + } + s << "}"; + return s; + } + + template <> + `nset` _arg<`nset`>(std::vector &args, unsigned idx, long long bound) { + throw std::invalid_argument("Not implemented"); // no syntax for nset values in the REPL + } + + >>> + } +} + +object classic_bft = { + relation quorum_intersection + private { + definition [quorum_intersection_def] quorum_intersection = forall Q1,Q2. nset.is_quorum(Q1) & nset.is_quorum(Q2) + -> exists N. well_behaved(N) & nset.member(N, Q1) & nset.member(N, Q2) # every two quorums have a well-behaved node in common + } +} + +trusted isolate accountable_bft = { + # this is our baseline assumption about quorums: + private { + property [max_2f_byzantine] nset.is_quorum(Q) -> exists N . well_behaved(N) & nset.member(N,Q) # every quorum has a well-behaved member + } +} diff --git a/cometbft/v0.38/spec/ivy-proofs/network_shim.ivy b/cometbft/v0.38/spec/ivy-proofs/network_shim.ivy new file mode 100644 index 00000000..ebc3a04f --- /dev/null +++ b/cometbft/v0.38/spec/ivy-proofs/network_shim.ivy @@ -0,0 +1,133 @@ +#lang ivy1.7 +# --- +# layout: page +# title: Network model and network shim +# --- + +# Here we define a network module, which is our model of the network, and a +# shim module that sits on top of the network and which, upon receiving a +# message, calls the appropriate protocol handler. + +include domain_model + +# Here we define an enumeration type for identifying the 3 different types of +# messages that nodes send. +object msg_kind = { # TODO: merge with step_t + type this = {proposal, prevote, precommit} +} + +# Here we define the type of messages `msg`. Its members are structs with the fields described below. +object msg = { + type this = struct { + m_kind : msg_kind, + m_src : node, + m_round : round, + m_value : value, + m_vround : round + } +} + +# This is our model of the network: +isolate net = { + + export action recv(dst:node,v:msg) + action send(src:node,dst:node,v:msg) + # Note that the `recv` action is exported, meaning that it can be called + # non-deterministically by the environment any time it is enabled. In other + # words, a packet that is in flight can be received at any time. In this + # sense, the network is fully asynchronous. Moreover, there is no + # requirement that a given message will be received at all. + + # The state of the network consists of all the packets that have been + # sent so far, along with their destination. + relation sent(V:msg, N:node) + + after init { + sent(V, N) := false + } + + before send { + sent(v,dst) := true + } + + before recv { + require sent(v,dst) # only sent messages can be received. + } +} + +# The network shim sits on top of the network and, upon receiving a message, +# calls the appropriate protocol handler. It also exposes a `broadcast` action +# that sends to all nodes. + +isolate shim = { + + # In order not repeat the same code for each handler, we use a handler + # module parameterized by the type of message it will handle. Below we + # instantiate this module for the 3 types of messages of Tendermint + module handler(p_kind) = { + action handle(dst:node,m:msg) + object spec = { + before handle { + assert sent(m,dst) & m.m_kind = p_kind + } + } + } + + instance proposal_handler : handler(msg_kind.proposal) + instance prevote_handler : handler(msg_kind.prevote) + instance precommit_handler : handler(msg_kind.precommit) + + relation sent(M:msg,N:node) + + action broadcast(src:node,m:msg) + action send(src:node,dst:node,m:msg) + + specification { + after init { + sent(M,D) := false; + } + before broadcast { + sent(m,D) := true + } + before send { + sent(m,dst) := true + } + } + + # Here we give an implementation of it that satisfies its specification: + implementation { + + implement net.recv(dst:node,m:msg) { + + if m.m_kind = msg_kind.proposal { + call proposal_handler.handle(dst,m) + } + else if m.m_kind = msg_kind.prevote { + call prevote_handler.handle(dst,m) + } + else if m.m_kind = msg_kind.precommit { + call precommit_handler.handle(dst,m) + } + } + + implement broadcast { # broadcast sends to all nodes, including the sender. + var iter := node.iter.create(0); + while ~iter.is_end + invariant net.sent(M,D) -> sent(M,D) + { + var n := iter.val; + call net.send(src,n,m); + iter := iter.next; + } + } + + implement send { + call net.send(src,dst,m) + } + + private { + invariant net.sent(M,D) -> sent(M,D) + } + } + +} with net, node # to prove that the shim implementation satisfies the shim specification, we rely on the specification of net and node. diff --git a/cometbft/v0.38/spec/ivy-proofs/output/.gitignore b/cometbft/v0.38/spec/ivy-proofs/output/.gitignore new file mode 100644 index 00000000..5e7d2734 --- /dev/null +++ b/cometbft/v0.38/spec/ivy-proofs/output/.gitignore @@ -0,0 +1,4 @@ +# Ignore everything in this directory +* +# Except this file +!.gitignore diff --git a/cometbft/v0.38/spec/ivy-proofs/tendermint.ivy b/cometbft/v0.38/spec/ivy-proofs/tendermint.ivy new file mode 100644 index 00000000..b7678bef --- /dev/null +++ b/cometbft/v0.38/spec/ivy-proofs/tendermint.ivy @@ -0,0 +1,420 @@ +#lang ivy1.7 +# --- +# layout: page +# title: Specification of Tendermint in Ivy +# --- + +# This specification closely follows the pseudo-code given in "The latest +# gossip on BFT consensus" by E. Buchman, J. Kwon, Z. Milosevic +# + +include domain_model +include network_shim + +# We model the Tendermint protocol as an Ivy object. Like in Object-Oriented +# Programming, the basic structuring unit in Ivy is the object. Objects have +# internal state and actions (i.e. methods in OO parlance) that modify their +# state. We model Tendermint as an object whose actions represent steps taken +# by individual nodes in the protocol. Actions in Ivy can have preconditions, +# and a valid execution is a sequence of actions whose preconditions are all +# satisfied in the state in which they are called. + +# For technical reasons, we define below a `tendermint` module instead of an +# object. Ivy modules are a little bit like classes in OO programs, and like +# classes they can be instantiated to obtain objects. To instantiate the +# `tendermint` module, we must provide an abstract-protocol object. This allows +# us to use different abstract-protocol objects for different parts of the +# proof, and to do so without too much notational burden (we could have used +# Ivy monitors, but then we would need to prefix every variable name by the +# name of the object containing it, which clutters things a bit compared to the +# approach we took). + +# The abstract-protocol object is called by the resulting tendermint object so +# as to run the abstract protocol alongside the concrete protocol. This allows +# us to transfer properties proved of the abstract protocol to the concrete +# protocol, as follows. First, we prove that running the abstract protocol in +# this way results in a valid execution of the abstract protocol. This is done +# by checking that all preconditions of the abstract actions are satisfied at +# their call sites. Second, we establish a relation between abstract state and +# concrete state (in the form of invariants of the resulting, two-object +# transition system) that allow us to transfer properties proved in the +# abstract protocol to the concrete protocol (for example, we prove that any +# decision made in the Tendermint protocol is also made in the abstract +# protocol; if the abstract protocol satisfies the agreement property, this +# allows us to conclude that the Tendermint protocol also does). + +# The abstract protocol object that we will use is always the same, and only +# the abstract properties that we prove about it change in the different +# instantiations of the `tendermint` module. Thus we provide common invariants +# that a) allow to prove that the abstract preconditions are met, and b) +# provide a refinement relation (see end of the module) relating the state of +# Tendermint to the state of the abstract protocol. + +# In the model, Byzantine nodes can send whatever messages they want, except +# that they cannot forge sender identities. This reflects the fact that, in +# practice, nodes use public key cryptography to sign their messages. + +# Finally, note that the observations that serve to adjudicate misbehavior are +# defined only in the abstract protocol (they happen in the abstract actions). + +module tendermint(abstract_protocol) = { + + # the initial value of a node: + function init_val(N:node): value + + # the three type of steps + object step_t = { + type this = {propose, prevote, precommit} + } # refer to those e.g. as step_t.propose + + object server(n:node) = { + + # the current round of a node + individual round_p: round + + individual step: step_t + + individual decision: value + + individual lockedValue: value + individual lockedRound: round + + individual validValue: value + individual validRound: round + + + relation done_l34(R:round) + relation done_l36(R:round, V:value) + relation done_l47(R:round) + + # variables for scheduling request + relation propose_timer_scheduled(R:round) + relation prevote_timer_scheduled(R:round) + relation precommit_timer_scheduled(R:round) + + relation _recved_proposal(Sender:node, R:round, V:value, VR:round) + relation _recved_prevote(Sender:node, R:round, V:value) + relation _recved_precommit(Sender:node, R:round, V:value) + + relation _has_started + + after init { + round_p := 0; + step := step_t.propose; + decision := value.nil; + + lockedValue := value.nil; + lockedRound := round.minus_one; + + validValue := value.nil; + validRound := round.minus_one; + + done_l34(R) := false; + done_l36(R, V) := false; + done_l47(R) := false; + + propose_timer_scheduled(R) := false; + prevote_timer_scheduled(R) := false; + precommit_timer_scheduled(R) := false; + + _recved_proposal(Sender, R, V, VR) := false; + _recved_prevote(Sender, R, V) := false; + _recved_precommit(Sender, R, V) := false; + + _has_started := false; + } + + action getValue returns (v:value) = { + v := init_val(n) + } + + export action start = { + require ~_has_started; + _has_started := true; + # line 10 + call startRound(0); + } + + # line 11-21 + action startRound(r:round) = { + # line 12 + round_p := r; + + # line 13 + step := step_t.propose; + + var proposal : value; + + # line 14 + if (proposers.get_proposer(r) = n) { + if validValue ~= value.nil { # line 15 + proposal := validValue; # line 16 + } else { + proposal := getValue(); # line 18 + }; + call broadcast_proposal(r, proposal, validRound); # line 19 + } else { + propose_timer_scheduled(r) := true; # line 21 + }; + + call abstract_protocol.l_11(n, r); + } + + # This action, as not exported, can only be called at specific call sites. + action broadcast_proposal(r:round, v:value, vr:round) = { + var m: msg; + m.m_kind := msg_kind.proposal; + m.m_src := n; + m.m_round := r; + m.m_value := v; + m.m_vround := vr; + call shim.broadcast(n,m); + } + + implement shim.proposal_handler.handle(msg:msg) { + _recved_proposal(msg.m_src, msg.m_round, msg.m_value, msg.m_vround) := true; + } + + # line 22-27 + export action l_22(v:value) = { + require _has_started; + require _recved_proposal(proposers.get_proposer(round_p), round_p, v, round.minus_one); + require step = step_t.propose; + + if (value.valid(v) & (lockedRound = round.minus_one | lockedValue = v)) { + call broadcast_prevote(round_p, v); # line 24 + call abstract_protocol.l_22(n, round_p, v); + } else { + call broadcast_prevote(round_p, value.nil); # line 26 + call abstract_protocol.l_22(n, round_p, value.nil); + }; + + # line 27 + step := step_t.prevote; + } + + # line 28-33 + export action l_28(r:round, v:value, vr:round, q:nset) = { + require _has_started; + require r = round_p; + require _recved_proposal(proposers.get_proposer(r), r, v, vr); + require nset.is_quorum(q); + require nset.member(N,q) -> _recved_prevote(N,vr,v); + require step = step_t.propose; + require vr >= 0 & vr < r; + + # line 29 + if (value.valid(v) & (lockedRound <= vr | lockedValue = v)) { + call broadcast_prevote(r, v); + } else { + call broadcast_prevote(r, value.nil); + }; + + call abstract_protocol.l_28(n,r,v,vr,q); + step := step_t.prevote; + } + + action broadcast_prevote(r:round, v:value) = { + var m: msg; + m.m_kind := msg_kind.prevote; + m.m_src := n; + m.m_round := r; + m.m_value := v; + call shim.broadcast(n,m); + } + + implement shim.prevote_handler.handle(msg:msg) { + _recved_prevote(msg.m_src, msg.m_round, msg.m_value) := true; + } + + # line 34-35 + export action l_34(r:round, q:nset) = { + require _has_started; + require round_p = r; + require nset.is_quorum(q); + require exists V . nset.member(N,q) -> _recved_prevote(N,r,V); + require step = step_t.prevote; + require ~done_l34(r); + done_l34(r) := true; + + prevote_timer_scheduled(r) := true; + } + + + # line 36-43 + export action l_36(r:round, v:value, q:nset) = { + require _has_started; + require r = round_p; + require exists VR . round.minus_one <= VR & VR < r & _recved_proposal(proposers.get_proposer(r), r, v, VR); + require nset.is_quorum(q); + require nset.member(N,q) -> _recved_prevote(N,r,v); + require value.valid(v); + require step = step_t.prevote | step = step_t.precommit; + + require ~done_l36(r,v); + done_l36(r, v) := true; + + if step = step_t.prevote { + lockedValue := v; # line 38 + lockedRound := r; # line 39 + call broadcast_precommit(r, v); # line 40 + step := step_t.precommit; # line 41 + call abstract_protocol.l_36(n, r, v, q); + }; + + validValue := v; # line 42 + validRound := r; # line 43 + } + + # line 44-46 + export action l_44(r:round, q:nset) = { + require _has_started; + require r = round_p; + require nset.is_quorum(q); + require nset.member(N,q) -> _recved_prevote(N,r,value.nil); + require step = step_t.prevote; + + call broadcast_precommit(r, value.nil); # line 45 + step := step_t.precommit; # line 46 + + call abstract_protocol.l_44(n, r, q); + } + + action broadcast_precommit(r:round, v:value) = { + var m: msg; + m.m_kind := msg_kind.precommit; + m.m_src := n; + m.m_round := r; + m.m_value := v; + call shim.broadcast(n,m); + } + + implement shim.precommit_handler.handle(msg:msg) { + _recved_precommit(msg.m_src, msg.m_round, msg.m_value) := true; + } + + + # line 47-48 + export action l_47(r:round, q:nset) = { + require _has_started; + require round_p = r; + require nset.is_quorum(q); + require nset.member(N,q) -> exists V . _recved_precommit(N,r,V); + require ~done_l47(r); + done_l47(r) := true; + + precommit_timer_scheduled(r) := true; + } + + + # line 49-54 + export action l_49_decide(r:round, v:value, q:nset) = { + require _has_started; + require exists VR . round.minus_one <= VR & VR < r & _recved_proposal(proposers.get_proposer(r), r, v, VR); + require nset.is_quorum(q); + require nset.member(N,q) -> _recved_precommit(N,r,v); + require decision = value.nil; + + if value.valid(v) { + decision := v; + # MORE for next height + call abstract_protocol.decide(n, r, v, q); + } + } + + # line 55-56 + export action l_55(r:round, b:nset) = { + require _has_started; + require nset.is_blocking(b); + require nset.member(N,b) -> exists VR . round.minus_one <= VR & VR < r & exists V . _recved_proposal(N,r,V,VR) | _recved_prevote(N,r,V) | _recved_precommit(N,r,V); + require r > round_p; + call startRound(r); # line 56 + } + + # line 57-60 + export action onTimeoutPropose(r:round) = { + require _has_started; + require propose_timer_scheduled(r); + require r = round_p; + require step = step_t.propose; + call broadcast_prevote(r,value.nil); + step := step_t.prevote; + + call abstract_protocol.l_57(n,r); + + propose_timer_scheduled(r) := false; + } + + # line 61-64 + export action onTimeoutPrevote(r:round) = { + require _has_started; + require prevote_timer_scheduled(r); + require r = round_p; + require step = step_t.prevote; + call broadcast_precommit(r,value.nil); + step := step_t.precommit; + + call abstract_protocol.l_61(n,r); + + prevote_timer_scheduled(r) := false; + } + + # line 65-67 + export action onTimeoutPrecommit(r:round) = { + require _has_started; + require precommit_timer_scheduled(r); + require r = round_p; + call startRound(round.incr(r)); + + precommit_timer_scheduled(r) := false; + } + +# The Byzantine actions +# --------------------- + +# Byzantine nodes can send whatever they want, but they cannot send +# messages on behalf of well-behaved nodes. In practice this is implemented +# using cryptography (e.g. public-key cryptography). + + export action byzantine_send(m:msg, dst:node) = { + require ~well_behaved(n); + require ~well_behaved(m.m_src); # cannot forge the identity of well-behaved nodes + call shim.send(n,dst,m); + } + +# Byzantine nodes can also report fake observations, as defined in the abstract protocol. + export action fake_observations = { + call abstract_protocol.misbehave + } + +# Invariants +# ---------- + +# We provide common invariants that a) allow to prove that the abstract +# preconditions are met, and b) provide a refinement relation. + + + specification { + + invariant 0 <= round_p + invariant abstract_protocol.left_round(n,R) <-> R < round_p + + invariant lockedRound ~= round.minus_one -> forall R,V . abstract_protocol.locked(n,R,V) <-> R <= lockedRound & lockedValue = V + invariant lockedRound = round.minus_one -> forall R,V . ~abstract_protocol.locked(n,R,V) + + invariant forall M:msg . well_behaved(M.m_src) & M.m_kind = msg_kind.prevote & shim.sent(M,N) -> abstract_protocol.prevoted(M.m_src,M.m_round,M.m_value) + invariant well_behaved(N) & _recved_prevote(N,R,V) -> abstract_protocol.prevoted(N,R,V) + invariant forall M:msg . well_behaved(M.m_src) & M.m_kind = msg_kind.precommit & shim.sent(M,N) -> abstract_protocol.precommitted(M.m_src,M.m_round,M.m_value) + invariant well_behaved(N) & _recved_precommit(N,R,V) -> abstract_protocol.precommitted(N,R,V) + + invariant (step = step_t.prevote | step = step_t.propose) -> ~abstract_protocol.precommitted(n,round_p,V) + invariant step = step_t.propose -> ~abstract_protocol.prevoted(n,round_p,V) + invariant step = step_t.prevote -> exists V . abstract_protocol.prevoted(n,round_p,V) + + invariant round_p < R -> ~(abstract_protocol.prevoted(n,R,V) | abstract_protocol.precommitted(n,R,V)) + invariant ~_has_started -> step = step_t.propose & ~(abstract_protocol.prevoted(n,R,V) | abstract_protocol.precommitted(n,R,V)) & round_p = 0 + + invariant decision ~= value.nil -> exists R . abstract_protocol.decided(n,R,decision) + } + } +} diff --git a/cometbft/v0.38/spec/ivy-proofs/tendermint_test.ivy b/cometbft/v0.38/spec/ivy-proofs/tendermint_test.ivy new file mode 100644 index 00000000..1299fc08 --- /dev/null +++ b/cometbft/v0.38/spec/ivy-proofs/tendermint_test.ivy @@ -0,0 +1,127 @@ +#lang ivy1.7 + +include tendermint +include abstract_tendermint + +isolate ghost_ = { + instantiate abstract_tendermint +} + +isolate protocol = { + instantiate tendermint(ghost_) # here we instantiate the parameter of the tendermint module with `ghost_`; however note that we don't extract any code for `ghost_` (it's not in the list of object in the extract, and it's thus sliced away). + implementation { + definition init_val(n:node) = <<< `n`%2 >>> + } + # attribute test = impl +} with ghost_, shim, value, round, proposers + +# Here we run a simple scenario that exhibits an execution in which nodes make +# a decision. We do this to rule out trivial modeling errors. + +# One option to check that this scenario is valid is to run it in Ivy's REPL. +# For this, first compile the scenario: +#```ivyc target=repl isolate=code trace=true tendermint_test.ivy +# Then, run the produced binary (e.g. for 4 nodes): +#``` ./tendermint_test 4 +# Finally, call the action: +#``` scenarios.scenario_1 +# Note that Ivy will check at runtime that all action preconditions are +# satisfied. For example, runing the scenario twice will cause a violation of +# the precondition of the `start` action, because a node cannot start twice +# (see `require ~_has_started` in action `start`). + +# Another possibility would be to run `ivy_check` on the scenario, but that +# does not seem to work at the moment. + +isolate scenarios = { + individual all:nset # will be used as parameter to actions requiring a quorum + + after init { + var iter := node.iter.create(0); + while ~iter.is_end + { + all := all.insert(iter.val); + iter := iter.next; + }; + assert nset.is_quorum(all); # we can also use asserts to make sure we are getting what we expect + } + + export action scenario_1 = { + # all nodes start: + var iter := node.iter.create(0); + while ~iter.is_end + { + call protocol.server.start(iter.val); + iter := iter.next; + }; + # all nodes receive the leader's proposal: + var m:msg; + m.m_kind := msg_kind.proposal; + m.m_src := 0; + m.m_round := 0; + m.m_value := 0; + m.m_vround := round.minus_one; + iter := node.iter.create(0); + while ~iter.is_end + { + call net.recv(iter.val,m); + iter := iter.next; + }; + # all nodes prevote: + iter := node.iter.create(0); + while ~iter.is_end + { + call protocol.server.l_22(iter.val,0); + iter := iter.next; + }; + # all nodes receive each other's prevote messages; + m.m_kind := msg_kind.prevote; + m.m_vround := 0; + iter := node.iter.create(0); + while ~iter.is_end + { + var iter2 := node.iter.create(0); # the sender + while ~iter2.is_end + { + m.m_src := iter2.val; + call net.recv(iter.val,m); + iter2 := iter2.next; + }; + iter := iter.next; + }; + # all nodes precommit: + iter := node.iter.create(0); + while ~iter.is_end + { + call protocol.server.l_36(iter.val,0,0,all); + iter := iter.next; + }; + # all nodes receive each other's pre-commits + m.m_kind := msg_kind.precommit; + iter := node.iter.create(0); + while ~iter.is_end + { + var iter2 := node.iter.create(0); # the sender + while ~iter2.is_end + { + m.m_src := iter2.val; + call net.recv(iter.val,m); + iter2 := iter2.next; + }; + iter := iter.next; + }; + # now all nodes can decide: + iter := node.iter.create(0); + while ~iter.is_end + { + call protocol.server.l_49_decide(iter.val,0,0,all); + iter := iter.next; + }; + } + + # TODO: add more scenarios + +} with round, node, proposers, value, nset, protocol, shim, net + +# extract code = protocol, shim, round, node +extract code = round, node, proposers, value, nset, protocol, shim, net, scenarios diff --git a/cometbft/v0.38/spec/light-client/Accountability.mdx b/cometbft/v0.38/spec/light-client/Accountability.mdx new file mode 100644 index 00000000..07179dd6 --- /dev/null +++ b/cometbft/v0.38/spec/light-client/Accountability.mdx @@ -0,0 +1,305 @@ +--- +order: 1 +parent: + title: Accountability + order: 4 +--- + +# Fork accountability + +## Problem Statement + +Tendermint consensus algorithm guarantees the following specifications for all heights: + +* agreement -- no two correct full nodes decide differently. +* validity -- the decided block satisfies the predefined predicate *valid()*. +* termination -- all correct full nodes eventually decide, + +If the faulty validators have less than 1/3 of voting power in the current validator set. In the case where this assumption +does not hold, each of the specification may be violated. + +The agreement property says that for a given height, any two correct validators that decide on a block for that height decide on the same block. That the block was indeed generated by the blockchain, can be verified starting from a trusted (genesis) block, and checking that all subsequent blocks are properly signed. + +However, faulty nodes may forge blocks and try to convince users (light clients) that the blocks had been correctly generated. In addition, Tendermint agreement might be violated in the case where 1/3 or more of the voting power belongs to faulty validators: Two correct validators decide on different blocks. The latter case motivates the term "fork": as Tendermint consensus also agrees on the next validator set, correct validators may have decided on disjoint next validator sets, and the chain branches into two or more partitions (possibly having faulty validators in common) and each branch continues to generate blocks independently of the other. + +We say that a fork is a case in which there are two commits for different blocks at the same height of the blockchain. The problem is to ensure that in those cases we are able to detect faulty validators (and not mistakenly accuse correct validators), and incentivize therefore validators to behave according to the protocol specification. + +**Conceptual Limit.** In order to prove misbehavior of a node, we have to show that the behavior deviates from correct behavior with respect to a given algorithm. Thus, an algorithm that detects misbehavior of nodes executing some algorithm *A* must be defined with respect to algorithm *A*. In our case, *A* is Tendermint consensus (+ other protocols in the infrastructure; e.g., Cosmos full nodes and the Light Client). If the consensus algorithm is changed/updated/optimized in the future, we have to check whether changes to the accountability algorithm are also required. All the discussions in this document are thus inherently specific to Tendermint consensus and the Light Client specification. + +**Q:** Should we distinguish agreement for validators and full nodes for agreement? The case where all correct validators agree on a block, but a correct full node decides on a different block seems to be slightly less severe that the case where two correct validators decide on different blocks. Still, if a contaminated full node becomes validator that may be problematic later on. Also it is not clear how gossiping is impaired if a contaminated full node is on a different branch. + +*Remark.* In the case 1/3 or more of the voting power belongs to faulty validators, also validity and termination can be broken. Termination can be broken if faulty processes just do not send the messages that are needed to make progress. Due to asynchrony, this is not punishable, because faulty validators can always claim they never received the messages that would have forced them to send messages. + +## The Misbehavior of Faulty Validators + +Forks are the result of faulty validators deviating from the protocol. In principle several such deviations can be detected without a fork actually occurring: + +1. double proposal: A faulty proposer proposes two different values (blocks) for the same height and the same round in Tendermint consensus. + +2. double signing: Tendermint consensus forces correct validators to prevote and precommit for at most one value per round. In case a faulty validator sends multiple prevote and/or precommit messages for different values for the same height/round, this is a misbehavior. + +3. lunatic validator: Tendermint consensus forces correct validators to prevote and precommit only for values *v* that satisfy *valid(v)*. If faulty validators prevote and precommit for *v* although *valid(v)=false* this is misbehavior. + +*Remark.* In isolation, Point 3 is an attack on validity (rather than agreement). However, the prevotes and precommits can then also be used to forge blocks. + +1. amnesia: Tendermint consensus has a locking mechanism. If a validator has some value v locked, then it can only prevote/precommit for v or nil. Sending prevote/precomit message for a different value v' (that is not nil) while holding lock on value v is misbehavior. + +2. spurious messages: In Tendermint consensus most of the message send instructions are guarded by threshold guards, e.g., one needs to receive *2f + 1* prevote messages to send precommit. Faulty validators may send precommit without having received the prevote messages. + +Independently of a fork happening, punishing this behavior might be important to prevent forks altogether. This should keep attackers from misbehaving: if less than 1/3 of the voting power is faulty, this misbehavior is detectable but will not lead to a safety violation. Thus, unless they have 1/3 or more (or in some cases more than 2/3) of the voting power attackers have the incentive to not misbehave. If attackers control too much voting power, we have to deal with forks, as discussed in this document. + +## Two types of forks + +* Fork-Full. Two correct validators decide on different blocks for the same height. Since also the next validator sets are decided upon, the correct validators may be partitioned to participate in two distinct branches of the forked chain. + +As in this case we have two different blocks (both having the same right/no right to exist), a central system invariant (one block per height decided by correct validators) is violated. As full nodes are contaminated in this case, the contamination can spread also to light clients. However, even without breaking this system invariant, light clients can be subject to a fork: + +* Fork-Light. All correct validators decide on the same block for height *h*, but faulty processes (validators or not), forge a different block for that height, in order to fool users (who use the light client). + +# Attack scenarios + +## On-chain attacks + +### Equivocation (one round) + +There are several scenarios in which forks might happen. The first is double signing within a round. + +* F1. Equivocation: faulty validators sign multiple vote messages (prevote and/or precommit) for different values *during the same round r* at a given height h. + +### Flip-flopping + +Tendermint consensus implements a locking mechanism: If a correct validator *p* receives proposal for value v and *2f + 1* prevotes for a value *id(v)* in round *r*, it locks *v* and remembers *r*. In this case, *p* also sends a precommit message for *id(v)*, which later may serve as proof that *p* locked *v*. +In subsequent rounds, *p* only sends prevote messages for a value it had previously locked. However, it is possible to change the locked value if in a future round *r' > r*, if the process receives proposal and *2f + 1* prevotes for a different value *v'*. In this case, *p* could send a prevote/precommit for *id(v')*. This algorithmic feature can be exploited in two ways: + +* F2. Faulty Flip-flopping (Amnesia): faulty validators precommit some value *id(v)* in round *r* (value *v* is locked in round *r*) and then prevote for different value *id(v')* in higher round *r' > r* without previously correctly unlocking value *v*. In this case faulty processes "forget" that they have locked value *v* and prevote some other value in the following rounds. +Some correct validators might have decided on *v* in *r*, and other correct validators decide on *v'* in *r'*. Here we can have branching on the main chain (Fork-Full). + +* F3. Correct Flip-flopping (Back to the past): There are some precommit messages signed by (correct) validators for value *id(v)* in round *r*. Still, *v* is not decided upon, and all processes move on to the next round. Then correct validators (correctly) lock and decide a different value *v'* in some round *r' > r*. And the correct validators continue; there is no branching on the main chain. +However, faulty validators may use the correct precommit messages from round *r* together with a posteriori generated faulty precommit messages for round *r* to forge a block for a value that was not decided on the main chain (Fork-Light). + +## Off-chain attacks + +F1-F3 may contaminate the state of full nodes (and even validators). Contaminated (but otherwise correct) full nodes may thus communicate faulty blocks to light clients. +Similarly, without actually interfering with the main chain, we can have the following: + +* F4. Phantom validators: faulty validators vote (sign prevote and precommit messages) in heights in which they are not part of the validator sets (at the main chain). + +* F5. Lunatic validator: faulty validator that sign vote messages to support (arbitrary) application state that is different from the application state that resulted from valid state transitions. + +## Types of victims + +We consider three types of potential attack victims: + +* FN: full node +* LCS: light client with sequential header verification +* LCB: light client with bisection based header verification + +F1 and F2 can be used by faulty validators to actually create multiple branches on the blockchain. That means that correctly operating full nodes decide on different blocks for the same height. Until a fork is detected locally by a full node (by receiving evidence from others or by some other local check that fails), the full node can spread corrupted blocks to light clients. + +*Remark.* If full nodes take a branch different from the one taken by the validators, it may be that the liveness of the gossip protocol may be affected. We should eventually look at this more closely. However, as it does not influence safety it is not a primary concern. + +F3 is similar to F1, except that no two correct validators decide on different blocks. It may still be the case that full nodes become affected. + +In addition, without creating a fork on the main chain, light clients can be contaminated by more than a third of validators that are faulty and sign a forged header +F4 cannot fool correct full nodes as they know the current validator set. Similarly, LCS know who the validators are. Hence, F4 is an attack against LCB that do not necessarily know the complete prefix of headers (Fork-Light), as they trust a header that is signed by at least one correct validator (trusting period method). + +The following table gives an overview of how the different attacks may affect different nodes. F1-F3 are *on-chain* attacks so they can corrupt the state of full nodes. Then if a light client (LCS or LCB) contacts a full node to obtain headers (or blocks), the corrupted state may propagate to the light client. + +F4 and F5 are *off-chain*, that is, these attacks cannot be used to corrupt the state of full nodes (which have sufficient knowledge on the state of the chain to not be fooled). + +| Attack | FN | LCS | LCB | +|:------:|:------:|:------:|:------:| +| F1 | direct | FN | FN | +| F2 | direct | FN | FN | +| F3 | direct | FN | FN | +| F4 | | | direct | +| F5 | | | direct | + +**Q:** Light clients are more vulnerable than full nodes, because the former do only verify headers but do not execute transactions. What kind of certainty is gained by a full node that executes a transaction? + +As a full node verifies all transactions, it can only be +contaminated by an attack if the blockchain itself violates its invariant (one block per height), that is, in case of a fork that leads to branching. + +## Detailed Attack Scenarios + +### Equivocation based attacks + +In case of equivocation based attacks, faulty validators sign multiple votes (prevote and/or precommit) in the same +round of some height. This attack can be executed on both full nodes and light clients. It requires 1/3 or more of voting power to be executed. + +#### Scenario 1: Equivocation on the main chain + +Validators: + +* CA - a set of correct validators with less than 1/3 of the voting power +* CB - a set of correct validators with less than 1/3 of the voting power +* CA and CB are disjoint +* F - a set of faulty validators with 1/3 or more voting power + +Observe that this setting violates the Cosmos failure model. + +Execution: + +* A faulty proposer proposes block A to CA +* A faulty proposer proposes block B to CB +* Validators from the set CA and CB prevote for A and B, respectively. +* Faulty validators from the set F prevote both for A and B. +* The faulty prevote messages + * for A arrive at CA long before the B messages + * for B arrive at CB long before the A messages +* Therefore correct validators from set CA and CB will observe +more than 2/3 of prevotes for A and B and precommit for A and B, respectively. +* Faulty validators from the set F precommit both values A and B. +* Thus, we have more than 2/3 commits for both A and B. + +Consequences: + +* Creating evidence of misbehavior is simple in this case as we have multiple messages signed by the same faulty processes for different values in the same round. + +* We have to ensure that these different messages reach a correct process (full node, monitor?), which can submit evidence. + +* This is an attack on the full node level (Fork-Full). +* It extends also to the light clients, +* For both we need a detection and recovery mechanism. + +#### Scenario 2: Equivocation to a light client (LCS) + +Validators: + +* a set F of faulty validators with more than 2/3 of the voting power. + +Execution: + +* for the main chain F behaves nicely +* F coordinates to sign a block B that is different from the one on the main chain. +* the light clients obtains B and trusts at as it is signed by more than 2/3 of the voting power. + +Consequences: + +Once equivocation is used to attack light client it opens space +for different kind of attacks as application state can be diverged in any direction. For example, it can modify validator set such that it contains only validators that do not have any stake bonded. Note that after a light client is fooled by a fork, that means that an attacker can change application state and validator set arbitrarily. + +In order to detect such (equivocation-based attack), the light client would need to cross check its state with some correct validator (or to obtain a hash of the state from the main chain using out of band channels). + +*Remark.* The light client would be able to create evidence of misbehavior, but this would require to pull potentially a lot of data from correct full nodes. Maybe we need to figure out different architecture where a light client that is attacked will push all its data for the current unbonding period to a correct node that will inspect this data and submit corresponding evidence. There are also architectures that assumes a special role (sometimes called fisherman) whose goal is to collect as much as possible useful data from the network, to do analysis and create evidence transactions. That functionality is outside the scope of this document. + +*Remark.* The difference between LCS and LCB might only be in the amount of voting power needed to convince light client about arbitrary state. In case of LCB where security threshold is at minimum, an attacker can arbitrarily modify application state with 1/3 or more of voting power, while in case of LCS it requires more than 2/3 of the voting power. + +### Flip-flopping: Amnesia based attacks + +In case of amnesia, faulty validators lock some value *v* in some round *r*, and then vote for different value *v'* in higher rounds without correctly unlocking value *v*. This attack can be used both on full nodes and light clients. + +#### Scenario 3: At most 2/3 of faults + +Validators: + +* a set F of faulty validators with 1/3 or more but at most 2/3 of the voting power +* a set C of correct validators + +Execution: + +* Faulty validators commit (without exposing it on the main chain) a block A in round *r* by collecting more than 2/3 of the + voting power (containing correct and faulty validators). +* All validators (correct and faulty) reach a round *r' > r*. +* Some correct validators in C do not lock any value before round *r'*. +* The faulty validators in F deviate from Tendermint consensus by ignoring that they locked A in *r*, and propose a different block B in *r'*. +* As the validators in C that have not locked any value find B acceptable, they accept the proposal for B and commit a block B. + +*Remark.* In this case, the more than 1/3 of faulty validators do not need to commit an equivocation (F1) as they only vote once per round in the execution. + +If a light client is attacked using this attack with 1/3 or more of voting power (and less than 2/3), the attacker cannot change the application state arbitrarily. Rather, the attacker is limited to a state a correct validator finds acceptable: In the execution above, correct validators still find the value acceptable, however, the block the light client trusts deviates from the one on the main chain. + +#### Scenario 4: More than 2/3 of faults + +In case there is an attack with more than 2/3 of the voting power, an attacker can arbitrarily change application state. + +Validators: + +* a set F1 of faulty validators with 1/3 or more of the voting power +* a set F2 of faulty validators with less than 1/3 of the voting power + +Execution + +* Similar to Scenario 3 (however, messages by correct validators are not needed) +* The faulty validators in F1 lock value A in round *r* +* They sign a different value in follow-up rounds +* F2 does not lock A in round *r* + +Consequences: + +* The validators in F1 will be detectable by the fork accountability mechanisms. +* The validators in F2 cannot be detected using this mechanism. +Only in case they signed something which conflicts with the application this can be used against them. Otherwise, they do not do anything incorrect. + +**Q:** do we need to define a special kind of attack for the case where a validator sign arbitrarily state? It seems that detecting such attack requires a different mechanism that would require as an evidence a sequence of blocks that led to that state. This might be very tricky to implement. + +### Back to the past + +In this kind of attack, faulty validators take advantage of the fact that they did not sign messages in some of the past rounds. Due to the asynchronous network in which Tendermint operates, we cannot easily differentiate between such an attack and delayed message. This kind of attack can be used at both full nodes and light clients. + +#### Scenario 5 + +Validators: + +* C1 - a set of correct validators with over 1/3 of the voting power +* C2 - a set of correct validators with 1/3 of the voting power +* C1 and C2 are disjoint +* F - a set of faulty validators with less than 1/3 voting power +* one additional faulty process *q* +* F and *q* violate the Cosmos failure model. + +Execution: + +* in a round *r* of height *h* we have C1 precommitting a value A, +* C2 precommits nil, +* F does not send any message +* *q* precommits nil. +* In some round *r' > r*, F and *q* and C2 commit some other value B different from A. +* F and *fp* "go back to the past" and sign precommit message for value A in round *r*. +* Together with precomit messages of C1 this is sufficient for a commit for value A. + +Consequences: + +* Only a single faulty validator that previously precommited nil did equivocation, while the other 1/3 of faulty validators actually executed an attack that has exactly the same sequence of messages as part of amnesia attack. Detecting this kind of attack boil down to mechanisms for equivocation and amnesia. + +**Q:** should we keep this as a separate kind of attack? It seems that equivocation, amnesia and phantom validators are the only kind of attack we need to support and this gives us security also in other cases. This would not be surprising as equivocation and amnesia are attacks that followed from the protocol and phantom attack is not really an attack to Tendermint but more to the Cosmos Proof of Stake module. + +### Phantom validators + +In case of phantom validators, processes that are not part of the current validator set but are still bonded (as attack happen during their unbonding period) can be part of the attack by signing vote messages. This attack can be executed against both full nodes and light clients. + +#### Scenario 6 + +Validators: + +* F -- a set of faulty validators that are not part of the validator set on the main chain at height *h + k* + +Execution: + +* There is a fork, and there exist two different headers for height *h + k*, with different validator sets: + * VS2 on the main chain + * forged header VS2', signed by F (and others) + +* a light client has a trust in a header for height *h* (and the corresponding validator set VS1). +* As part of bisection header verification, it verifies the header at height *h + k* with new validator set VS2'. + +Consequences: + +* To detect this, a node needs to see both, the forged header and the canonical header from the chain. +* If this is the case, detecting these kind of attacks is easy as it just requires verifying if processes are signing messages in heights in which they are not part of the validator set. + +**Remark.** We can have phantom-validator-based attacks as a follow up of equivocation or amnesia based attack where forked state contains validators that are not part of the validator set at the main chain. In this case, they keep signing messages contributed to a forked chain (the wrong branch) although they are not part of the validator set on the main chain. This attack can also be used to attack full node during a period of time it is eclipsed. + +**Remark.** Phantom validator evidence has been removed from implementation as it was deemed, although possibly a plausible form of evidence, not relevant. Any attack on +the light client involving a phantom validator will have needed to be initiated by 1/3+ lunatic +validators that can forge a new validator set that includes the phantom validator. Only in +that case will the light client accept the phantom validators vote. We need only worry about +punishing the 1/3+ lunatic cabal, that is the root cause of the attack. + +### Lunatic validator + +Lunatic validator agrees to sign commit messages for arbitrary application state. It is used to attack light clients. +Note that detecting this behavior require application knowledge. Detecting this behavior can probably be done by +referring to the block before the one in which height happen. + +**Q:** can we say that in this case a validator declines to check if a proposed value is valid before voting for it? diff --git a/cometbft/v0.38/spec/light-client/Fork-Detection.mdx b/cometbft/v0.38/spec/light-client/Fork-Detection.mdx new file mode 100644 index 00000000..4bcd78cc --- /dev/null +++ b/cometbft/v0.38/spec/light-client/Fork-Detection.mdx @@ -0,0 +1,75 @@ +--- +order: 1 +parent: + title: Fork Detection + order: 2 +--- + +# Cosmos fork detection and IBC fork detection + +## Status + +This is a work in progress. +This directory captures the ongoing work and discussion on fork +detection both in the context of a Cosmos light node and in the +context of IBC. It contains the following files + +### [detection.md](./detection_003_reviewed.md) + +a draft of the light node fork detection including "proof of fork" + definition, that is, the data structure to submit evidence to full + nodes. + +### [discussions.md](./discussions.md) + +A collection of ideas and intuitions from recent discussions + +- the outcome of recent discussion +- a sketch of the light client supervisor to provide the context in + which fork detection happens +- a discussion about lightstore semantics + +### [req-ibc-detection.md](./req-ibc-detection.md) + +- a collection of requirements for fork detection in the IBC + context. In particular it contains a section "Required Changes in + ICS 007" with necessary updates to ICS 007 to support Cosmos + fork detection + +### [draft-functions.md](./draft-functions.md) + +In order to address the collected requirements, we started to sketch +some functions that we will need in the future when we specify in more +detail the + +- fork detections +- proof of fork generation +- proof of fork verification + +on the following components. + +- IBC on-chain components +- Relayer + +### TODOs + +We decided to merge the files while there are still open points to +address to record the current state an move forward. In particular, +the following points need to be addressed: + +- [https://github.com/informalsystems/tendermint-rs/pull/479#discussion_r466504876](https://github.com/informalsystems/tendermint-rs/pull/479#discussion_r466504876) + +- [https://github.com/informalsystems/tendermint-rs/pull/479#discussion_r466493900](https://github.com/informalsystems/tendermint-rs/pull/479#discussion_r466493900) + +- [https://github.com/informalsystems/tendermint-rs/pull/479#discussion_r466489045](https://github.com/informalsystems/tendermint-rs/pull/479#discussion_r466489045) + +- [https://github.com/informalsystems/tendermint-rs/pull/479#discussion_r466491471](https://github.com/informalsystems/tendermint-rs/pull/479#discussion_r466491471) + +Most likely we will write a specification on the light client +supervisor along the outcomes of + +- [https://github.com/informalsystems/tendermint-rs/pull/509](https://github.com/informalsystems/tendermint-rs/pull/509) + +that also addresses initialization + +- [https://github.com/tendermint/spec/issues/131](https://github.com/tendermint/spec/issues/131) diff --git a/cometbft/v0.38/spec/light-client/Light-Client-Specification.mdx b/cometbft/v0.38/spec/light-client/Light-Client-Specification.mdx new file mode 100644 index 00000000..19285d79 --- /dev/null +++ b/cometbft/v0.38/spec/light-client/Light-Client-Specification.mdx @@ -0,0 +1,205 @@ +--- +order: 1 +parent: + title: Light Client + order: 5 +--- + +# Light Client Specification + +This directory contains work-in-progress English and TLA+ specifications for the Light Client +protocol. Implementations of the light client can be found in +[Rust](https://github.com/informalsystems/tendermint-rs/tree/master/light-client) and +[Go](https://github.com/cometbft/cometbft/tree/v0.38.x/light). + +Light clients are assumed to be initialized once from a trusted source +with a trusted header and validator set. The light client +protocol allows a client to then securely update its trusted state by requesting and +verifying a minimal set of data from a network of full nodes (at least one of which is correct). + +The light client is decomposed into two main components: + +- [Commit Verification](#commit-verification) - verify signed headers and associated validator + set changes from a single full node, called primary +- [Attack Detection](#attack-detection) - verify commits across multiple full nodes (called secondaries) and detect conflicts (ie. the existence of a lightclient attack) + +In case a lightclient attack is detected, the lightclient submits evidence to a full node which is responsible for "accountability", that is, punishing attackers: + +- [Accountability](#accountability) - given evidence for an attack, compute a set of validators that are responsible for it. + +## Commit Verification + +The [English specification](verification/verification_001_published.md) describes the light client +commit verification problem in terms of the temporal properties +[LCV-DIST-SAFE.1](https://github.com/cometbft/cometbft/blob/v0.38.x/spec/light-client/verification/verification_001_published.md#lcv-dist-safe1) and +[LCV-DIST-LIVE.1](https://github.com/cometbft/cometbft/blob/v0.38.x/spec/light-client/verification/verification_001_published.md#lcv-dist-live1). +Commit verification is assumed to operate within the Cosmos Failure Model, where +2/3 of validators are correct for some time period and +validator sets can change arbitrarily at each height. + +A light client protocol is also provided, including all checks that +need to be performed on headers, commits, and validator sets +to satisfy the temporal properties - so a light client can continuously +synchronize with a blockchain. Clients can skip possibly +many intermediate headers by exploiting overlap in trusted and untrusted validator sets. +When there is not enough overlap, a bisection routine can be used to find a +minimal set of headers that do provide the required overlap. + +The [TLA+ specification ver. 001](verification/Lightclient_A_1.tla) +is a formal description of the +commit verification protocol executed by a client, including the safety and +termination, which can be model checked with Apalache. + +A more detailed TLA+ specification of +[Light client verification ver. 003](verification/Lightclient_003_draft.tla) +is currently under peer review. + +The `MC*.tla` files contain concrete parameters for the +[TLA+ specification](verification/Lightclient_A_1.tla), in order to do model checking. +For instance, [MC4_3_faulty.tla](verification/MC4_3_faulty.tla) contains the following parameters +for the nodes, heights, the trusting period, the clock drifts, +correctness of the primary node, and the ratio of the faulty processes: + +```tla +AllNodes == {"n1", "n2", "n3", "n4"} +TRUSTED_HEIGHT == 1 +TARGET_HEIGHT == 3 +TRUSTING_PERIOD == 1400 \* the trusting period in some time units +CLOCK_DRIFT = 10 \* how much we assume the local clock is drifting +REAL_CLOCK_DRIFT = 3 \* how much the local clock is actually drifting +IS_PRIMARY_CORRECT == FALSE +FAULTY_RATIO == <<1, 3>> \* < 1 / 3 faulty validators +``` + +To run a complete set of experiments, clone [apalache](https://github.com/informalsystems/apalache) and [apalache-tests](https://github.com/informalsystems/apalache-tests) into a directory `$DIR` and run the following commands: + +```sh +$DIR/apalache-tests/scripts/mk-run.py --memlimit 28 002bmc-apalache-ok.csv $DIR/apalache . out +./out/run-all.sh +``` + +After the experiments have finished, you can collect the logs by executing the following command: + +```sh +cd ./out +$DIR/apalache-tests/scripts/parse-logs.py --human . +``` + +All lines in `results.csv` should report `Deadlock`, which means that the algorithm +has terminated and no invariant violation was found. + +Similar to [002bmc-apalache-ok.csv](verification/002bmc-apalache-ok.csv), +file [003bmc-apalache-error.csv](verification/003bmc-apalache-error.csv) specifies +the set of experiments that should result in counterexamples: + +```sh +$DIR/apalache-tests/scripts/mk-run.py --memlimit 28 003bmc-apalache-error.csv $DIR/apalache . out +./out/run-all.sh +``` + +All lines in `results.csv` should report `Error`. + +The following table summarizes the experimental results for Light client verification +version 001. The TLA+ properties can be found in the +[TLA+ specification](verification/Lightclient_A_1.tla). + The experiments were run in an AWS instance equipped with 32GB +RAM and a 4-core Intel® Xeon® CPU E5-2686 v4 @ 2.30GHz CPU. +We write “`✗=k`” when a bug is reported at depth k, and “`✓<=k`” when +no bug is reported up to depth k. + +![Experimental results](experiments.png) + +The experimental results for version 003 are to be added. + +## Attack Detection + +The [English specification](detection/detection_003_reviewed.md) +defines light client attacks (and how they differ from blockchain +forks), and describes the problem of a light client detecting +these attacks by communicating with a network of full nodes, +where at least one is correct. + +The specification also contains a detection protocol that checks +whether the header obtained from the primary via the verification +protocol matches corresponding headers provided by the secondaries. +If this is not the case, the protocol analyses the verification traces +of the involved full nodes +and generates +[evidence](detection/detection_003_reviewed.md#cmbc-lc-evidence-data1) +of misbehavior that can be submitted to a full node so that +the faulty validators can be punished. + +The [TLA+ specification](detection/LCDetector_003_draft.tla) +is a formal description of the +detection protocol for two peers, including the safety and +termination, which can be model checked with Apalache. + +The `LCD_MC*.tla` files contain concrete parameters for the +[TLA+ specification](detection/LCDetector_003_draft.tla), +in order to run the model checker. +For instance, [LCD_MC4_4_faulty.tla](./verification/MC4_4_faulty.tla) +contains the following parameters +for the nodes, heights, the trusting period, the clock drifts, +correctness of the nodes, and the ratio of the faulty processes: + +```tla +AllNodes == {"n1", "n2", "n3", "n4"} +TRUSTED_HEIGHT == 1 +TARGET_HEIGHT == 3 +TRUSTING_PERIOD == 1400 \* the trusting period in some time units +CLOCK_DRIFT = 10 \* how much we assume the local clock is drifting +REAL_CLOCK_DRIFT = 3 \* how much the local clock is actually drifting +IS_PRIMARY_CORRECT == FALSE +IS_SECONDARY_CORRECT == FALSE +FAULTY_RATIO == <<1, 3>> \* < 1 / 3 faulty validators +``` + +To run a complete set of experiments, clone [apalache](https://github.com/informalsystems/apalache) and [apalache-tests](https://github.com/informalsystems/apalache-tests) into a directory `$DIR` and run the following commands: + +```sh +$DIR/apalache-tests/scripts/mk-run.py --memlimit 28 004bmc-apalache-ok.csv $DIR/apalache . out +./out/run-all.sh +``` + +After the experiments have finished, you can collect the logs by executing the following command: + +```sh +cd ./out +$DIR/apalache-tests/scripts/parse-logs.py --human . +``` + +All lines in `results.csv` should report `Deadlock`, which means that the algorithm +has terminated and no invariant violation was found. + +Similar to [004bmc-apalache-ok.csv](verification/004bmc-apalache-ok.csv), +file [005bmc-apalache-error.csv](verification/005bmc-apalache-error.csv) specifies +the set of experiments that should result in counterexamples: + +```sh +$DIR/apalache-tests/scripts/mk-run.py --memlimit 28 005bmc-apalache-error.csv $DIR/apalache . out +./out/run-all.sh +``` + +All lines in `results.csv` should report `Error`. + +The detailed experimental results are to be added soon. + +## Accountability + +The [English specification](attacks/isolate-attackers_002_reviewed.md) +defines the protocol that is executed on a full node upon receiving attack [evidence](detection/detection_003_reviewed.md#cmbc-lc-evidence-data1) from a lightclient. In particular, the protocol handles three types of attacks + +- lunatic +- equivocation +- amnesia + +We discussed in the [last part](attacks/isolate-attackers_002_reviewed.md#Part-III---Completeness) of the English specification +that the non-lunatic cases are defined by having the same validator set in the conflicting blocks. For these cases, +computer-aided analysis of [Tendermint Consensus in TLA+](./accountability/README.md) shows that equivocation and amnesia capture all non-lunatic attacks. + +The [TLA+ specification](attacks/Isolation_001_draft.tla) +is a formal description of the +protocol, including the safety property, which can be model checked with Apalache. + +Similar to the other specifications, [MC_5_3.tla](attacks/MC_5_3.tla) contains concrete parameters to run the model checker. The specification can be checked within seconds. + +[tendermint-accountability](./accountability/README.md) diff --git a/cometbft/v0.38/spec/light-client/assets/light-node-image.png b/cometbft/v0.38/spec/light-client/assets/light-node-image.png new file mode 100644 index 00000000..f0b93c6e Binary files /dev/null and b/cometbft/v0.38/spec/light-client/assets/light-node-image.png differ diff --git a/cometbft/v0.38/spec/light-client/experiments.png b/cometbft/v0.38/spec/light-client/experiments.png new file mode 100644 index 00000000..94166ffa Binary files /dev/null and b/cometbft/v0.38/spec/light-client/experiments.png differ diff --git a/cometbft/v0.38/spec/light-client/verification.mdx b/cometbft/v0.38/spec/light-client/verification.mdx new file mode 100644 index 00000000..7a243fbb --- /dev/null +++ b/cometbft/v0.38/spec/light-client/verification.mdx @@ -0,0 +1,584 @@ +--- +order: 1 +parent: + title: Verification + order: 2 +--- +# Core Verification + +## Problem statement + +We assume that the light client knows a (base) header `inithead` it trusts (by social consensus or because +the light client has decided to trust the header before). The goal is to check whether another header +`newhead` can be trusted based on the data in `inithead`. + +The correctness of the protocol is based on the assumption that `inithead` was generated by an instance of +Tendermint consensus. + +### Failure Model + +For the purpose of the following definitions we assume that there exists a function +`validators` that returns the corresponding validator set for the given hash. + +The light client protocol is defined with respect to the following failure model: + +Given a known bound `TRUSTED_PERIOD`, and a block `b` with header `h` generated at time `Time` +(i.e. `h.Time = Time`), a set of validators that hold more than 2/3 of the voting power +in `validators(b.Header.NextValidatorsHash)` is correct until time `b.Header.Time + TRUSTED_PERIOD`. + +*Assumption*: "correct" is defined w.r.t. realtime (some Newtonian global notion of time, i.e., wall time), +while `Header.Time` corresponds to the [BFT time](../../consensus/bft-time.md). In this note, we assume that clocks of correct processes +are synchronized (for example using NTP), and therefore there is bounded clock drift (`CLOCK_DRIFT`) between local clocks and +BFT time. More precisely, for every correct light client process and every `header.Time` (i.e. BFT Time, for a header correctly +generated by the Tendermint consensus), the following inequality holds: `Header.Time < now + CLOCK_DRIFT`, +where `now` corresponds to the system clock at the light client process. + +Furthermore, we assume that `TRUSTED_PERIOD` is (several) order of magnitude bigger than `CLOCK_DRIFT` (`TRUSTED_PERIOD >> CLOCK_DRIFT`), +as `CLOCK_DRIFT` (using NTP) is in the order of milliseconds and `TRUSTED_PERIOD` is in the order of weeks. + +We expect a light client process defined in this document to be used in the context in which there is some +larger period during which misbehaving validators can be detected and punished (we normally refer to it as `UNBONDING_PERIOD` +due to the "bonding" mechanism in modern proof of stake systems). Furthermore, we assume that +`TRUSTED_PERIOD < UNBONDING_PERIOD` and that they are normally of the same order of magnitude, for example +`TRUSTED_PERIOD = UNBONDING_PERIOD / 2`. + +The specification in this document considers an implementation of the light client under the Failure Model defined above. +Mechanisms like `fork accountability` and `evidence submission` are defined in the context of `UNBONDING_PERIOD` and +they incentivize validators to follow the protocol specification defined in this document. If they don't, +and we have 1/3 (or more) faulty validators, safety may be violated. Our approach then is +to *detect* these cases (after the fact), and take suitable repair actions (automatic and social). +This is discussed in document on [Fork accountability](../../consensus/light-client/accountability.md). + +The term "trusted" above indicates that the correctness of the protocol depends on +this assumption. It is in the responsibility of the user that runs the light client to make sure that the risk +of trusting a corrupted/forged `inithead` is negligible. + +*Remark*: This failure model might change to a hybrid version that takes heights into account in the future. + +### High Level Solution + +Upon initialization, the light client is given a header `inithead` it trusts (by +social consensus). When a light clients sees a new signed header `snh`, it has to decide whether to trust the new +header. Trust can be obtained by (possibly) the combination of three methods. + +1. **Uninterrupted sequence of headers.** Given a trusted header `h` and an untrusted header `h1`, +the light client trusts a header `h1` if it trusts all headers in between `h` and `h1`. + +2. **Trusted period.** Given a trusted header `h`, an untrusted header `h1 > h` and `TRUSTED_PERIOD` during which +the failure model holds, we can check whether at least one validator, that has been continuously correct +from `h.Time` until now, has signed `h1`. If this is the case, we can trust `h1`. + +3. **Bisection.** If a check according to 2. (trusted period) fails, the light client can try to +obtain a header `hp` whose height lies between `h` and `h1` in order to check whether `h` can be used to +get trust for `hp`, and `hp` can be used to get trust for `snh`. If this is the case we can trust `h1`; +if not, we continue recursively until either we found set of headers that can build (transitively) trust relation +between `h` and `h1`, or we failed as two consecutive headers don't verify against each other. + +## Definitions + +### Data structures + +In the following, only the details of the data structures needed for this specification are given. + + ```go + type Header struct { + Height int64 + Time Time // the chain time when the header (block) was generated + + LastBlockID BlockID // prev block info + ValidatorsHash []byte // hash of the validators for the current block + NextValidatorsHash []byte // hash of the validators for the next block + } + + type SignedHeader struct { + Header Header + Commit Commit // commit for the given header + } + + type ValidatorSet struct { + Validators []Validator + TotalVotingPower int64 + } + + type Validator struct { + Address Address // validator address (we assume validator's addresses are unique) + VotingPower int64 // validator's voting power + } + + type TrustedState { + SignedHeader SignedHeader + ValidatorSet ValidatorSet + } + ``` + +### Functions + +For the purpose of this light client specification, we assume that the Cosmos Full Node +exposes the following functions over RPC: + +```go + // returns signed header: Header with Commit, for the given height + func Commit(height int64) (SignedHeader, error) + + // returns validator set for the given height + func Validators(height int64) (ValidatorSet, error) +``` + +Furthermore, we assume the following auxiliary functions: + +```go + // returns true if the commit is for the header, ie. if it contains + // the correct hash of the header; otherwise false + func matchingCommit(header Header, commit Commit) bool + + // returns the set of validators from the given validator set that + // committed the block (that correctly signed the block) + // it assumes signature verification so it can be computationally expensive + func signers(commit Commit, validatorSet ValidatorSet) []Validator + + // returns the voting power the validators in v1 have according to their voting power in set v2 + // it does not assume signature verification + func votingPowerIn(v1 []Validator, v2 ValidatorSet) int64 + + // returns hash of the given validator set + func hash(v2 ValidatorSet) []byte +``` + +In the functions below we will be using `trustThreshold` as a parameter. For simplicity +we assume that `trustThreshold` is a float between `1/3` and `2/3` and we will not be checking it +in the pseudo-code. + +**VerifySingle.** The function `VerifySingle` attempts to validate given untrusted header and the corresponding validator sets +based on a given trusted state. It ensures that the trusted state is still within its trusted period, +and that the untrusted header is within assumed `clockDrift` bound of the passed time `now`. +Note that this function is not making external (RPC) calls to the full node; the whole logic is +based on the local (given) state. This function is supposed to be used by the IBC handlers. + +```go +func VerifySingle(untrustedSh SignedHeader, + untrustedVs ValidatorSet, + untrustedNextVs ValidatorSet, + trustedState TrustedState, + trustThreshold float, + trustingPeriod Duration, + clockDrift Duration, + now Time) (TrustedState, error) { + + if untrustedSh.Header.Time > now + clockDrift { + return (trustedState, ErrInvalidHeaderTime) + } + + trustedHeader = trustedState.SignedHeader.Header + if !isWithinTrustedPeriod(trustedHeader, trustingPeriod, now) { + return (state, ErrHeaderNotWithinTrustedPeriod) + } + + // we assume that time it takes to execute verifySingle function + // is several order of magnitudes smaller than trustingPeriod + error = verifySingle( + trustedState, + untrustedSh, + untrustedVs, + untrustedNextVs, + trustThreshold) + + if error != nil return (state, error) + + // the untrusted header is now trusted + newTrustedState = TrustedState(untrustedSh, untrustedNextVs) + return (newTrustedState, nil) +} + +// return true if header is within its light client trusted period; otherwise returns false +func isWithinTrustedPeriod(header Header, + trustingPeriod Duration, + now Time) bool { + + return header.Time + trustedPeriod > now +} +``` + +Note that in case `VerifySingle` returns without an error (untrusted header +is successfully verified) then we have a guarantee that the transition of the trust +from `trustedState` to `newTrustedState` happened during the trusted period of +`trustedState.SignedHeader.Header`. + +TODO: Explain what happens in case `VerifySingle` returns with an error. + +**verifySingle.** The function `verifySingle` verifies a single untrusted header +against a given trusted state. It includes all validations and signature verification. +It is not publicly exposed since it does not check for header expiry (time constraints) +and hence it's possible to use it incorrectly. + +```go +func verifySingle(trustedState TrustedState, + untrustedSh SignedHeader, + untrustedVs ValidatorSet, + untrustedNextVs ValidatorSet, + trustThreshold float) error { + + untrustedHeader = untrustedSh.Header + untrustedCommit = untrustedSh.Commit + + trustedHeader = trustedState.SignedHeader.Header + trustedVs = trustedState.ValidatorSet + + if trustedHeader.Height >= untrustedHeader.Height return ErrNonIncreasingHeight + if trustedHeader.Time >= untrustedHeader.Time return ErrNonIncreasingTime + + // validate the untrusted header against its commit, vals, and next_vals + error = validateSignedHeaderAndVals(untrustedSh, untrustedVs, untrustedNextVs) + if error != nil return error + + // check for adjacent headers + if untrustedHeader.Height == trustedHeader.Height + 1 { + if trustedHeader.NextValidatorsHash != untrustedHeader.ValidatorsHash { + return ErrInvalidAdjacentHeaders + } + } else { + error = verifyCommitTrusting(trustedVs, untrustedCommit, untrustedVs, trustThreshold) + if error != nil return error + } + + // verify the untrusted commit + return verifyCommitFull(untrustedVs, untrustedCommit) +} + +// returns nil if header and validator sets are consistent; otherwise returns error +func validateSignedHeaderAndVals(signedHeader SignedHeader, vs ValidatorSet, nextVs ValidatorSet) error { + header = signedHeader.Header + if hash(vs) != header.ValidatorsHash return ErrInvalidValidatorSet + if hash(nextVs) != header.NextValidatorsHash return ErrInvalidNextValidatorSet + if !matchingCommit(header, signedHeader.Commit) return ErrInvalidCommitValue + return nil +} + +// returns nil if at least single correst signer signed the commit; otherwise returns error +func verifyCommitTrusting(trustedVs ValidatorSet, + commit Commit, + untrustedVs ValidatorSet, + trustLevel float) error { + + totalPower := trustedVs.TotalVotingPower + signedPower := votingPowerIn(signers(commit, untrustedVs), trustedVs) + + // check that the signers account for more than max(1/3, trustLevel) of the voting power + // this ensures that there is at least single correct validator in the set of signers + if signedPower < max(1/3, trustLevel) * totalPower return ErrInsufficientVotingPower + return nil +} + +// returns nil if commit is signed by more than 2/3 of voting power of the given validator set +// return error otherwise +func verifyCommitFull(vs ValidatorSet, commit Commit) error { + totalPower := vs.TotalVotingPower; + signedPower := votingPowerIn(signers(commit, vs), vs) + + // check the signers account for +2/3 of the voting power + if signedPower * 3 <= totalPower * 2 return ErrInvalidCommit + return nil +} +``` + +**VerifyHeaderAtHeight.** The function `VerifyHeaderAtHeight` captures high level +logic, i.e., application call to the light client module to download and verify header +for some height. + +```go +func VerifyHeaderAtHeight(untrustedHeight int64, + trustedState TrustedState, + trustThreshold float, + trustingPeriod Duration, + clockDrift Duration) (TrustedState, error)) { + + trustedHeader := trustedState.SignedHeader.Header + + now := System.Time() + if !isWithinTrustedPeriod(trustedHeader, trustingPeriod, now) { + return (trustedState, ErrHeaderNotWithinTrustedPeriod) + } + + newTrustedState, err := VerifyBisection(untrustedHeight, + trustedState, + trustThreshold, + trustingPeriod, + clockDrift, + now) + + if err != nil return (trustedState, err) + + now = System.Time() + if !isWithinTrustedPeriod(trustedHeader, trustingPeriod, now) { + return (trustedState, ErrHeaderNotWithinTrustedPeriod) + } + + return (newTrustedState, err) +} +``` + +Note that in case `VerifyHeaderAtHeight` returns without an error (untrusted header +is successfully verified) then we have a guarantee that the transition of the trust +from `trustedState` to `newTrustedState` happened during the trusted period of +`trustedState.SignedHeader.Header`. + +In case `VerifyHeaderAtHeight` returns with an error, then either (i) the full node we are talking to is faulty +or (ii) the trusted header has expired (it is outside its trusted period). In case (i) the full node is faulty so +light client should disconnect and reinitialize with new peer. In the case (ii) as the trusted header has expired, +we need to reinitialize light client with a new trusted header (that is within its trusted period), +but we don't necessarily need to disconnect from the full node we are talking to (as we haven't observed full node misbehavior in this case). + +**VerifyBisection.** The function `VerifyBisection` implements +recursive logic for checking if it is possible building trust +relationship between `trustedState` and untrusted header at the given height over +finite set of (downloaded and verified) headers. + +```go +func VerifyBisection(untrustedHeight int64, + trustedState TrustedState, + trustThreshold float, + trustingPeriod Duration, + clockDrift Duration, + now Time) (TrustedState, error) { + + untrustedSh, error := Commit(untrustedHeight) + if error != nil return (trustedState, ErrRequestFailed) + + untrustedHeader = untrustedSh.Header + + // note that we pass now during the recursive calls. This is fine as + // all other untrusted headers we download during recursion will be + // for a smaller heights, and therefore should happen before. + if untrustedHeader.Time > now + clockDrift { + return (trustedState, ErrInvalidHeaderTime) + } + + untrustedVs, error := Validators(untrustedHeight) + if error != nil return (trustedState, ErrRequestFailed) + + untrustedNextVs, error := Validators(untrustedHeight + 1) + if error != nil return (trustedState, ErrRequestFailed) + + error = verifySingle( + trustedState, + untrustedSh, + untrustedVs, + untrustedNextVs, + trustThreshold) + + if fatalError(error) return (trustedState, error) + + if error == nil { + // the untrusted header is now trusted. + newTrustedState = TrustedState(untrustedSh, untrustedNextVs) + return (newTrustedState, nil) + } + + // at this point in time we need to do bisection + pivotHeight := ceil((trustedHeader.Height + untrustedHeight) / 2) + + error, newTrustedState = VerifyBisection(pivotHeight, + trustedState, + trustThreshold, + trustingPeriod, + clockDrift, + now) + if error != nil return (newTrustedState, error) + + return VerifyBisection(untrustedHeight, + newTrustedState, + trustThreshold, + trustingPeriod, + clockDrift, + now) +} + +func fatalError(err) bool { + return err == ErrHeaderNotWithinTrustedPeriod OR + err == ErrInvalidAdjacentHeaders OR + err == ErrNonIncreasingHeight OR + err == ErrNonIncreasingTime OR + err == ErrInvalidValidatorSet OR + err == ErrInvalidNextValidatorSet OR + err == ErrInvalidCommitValue OR + err == ErrInvalidCommit +} +``` + +### The case `untrustedHeader.Height < trustedHeader.Height` + +In the use case where someone tells the light client that application data that is relevant for it +can be read in the block of height `k` and the light client trusts a more recent header, we can use the +hashes to verify headers "down the chain." That is, we iterate down the heights and check the hashes in each step. + +*Remark.* For the case were the light client trusts two headers `i` and `j` with `i < k < j`, we should +discuss/experiment whether the forward or the backward method is more effective. + +```go +func VerifyHeaderBackwards(trustedHeader Header, + untrustedHeader Header, + trustingPeriod Duration, + clockDrift Duration) error { + + if untrustedHeader.Height >= trustedHeader.Height return ErrErrNonDecreasingHeight + if untrustedHeader.Time >= trustedHeader.Time return ErrNonDecreasingTime + + now := System.Time() + if !isWithinTrustedPeriod(trustedHeader, trustingPeriod, now) { + return ErrHeaderNotWithinTrustedPeriod + } + + old := trustedHeader + for i := trustedHeader.Height - 1; i > untrustedHeader.Height; i-- { + untrustedSh, error := Commit(i) + if error != nil return ErrRequestFailed + + if (hash(untrustedSh.Header) != old.LastBlockID.Hash) { + return ErrInvalidAdjacentHeaders + } + + old := untrustedSh.Header + } + + if hash(untrustedHeader) != old.LastBlockID.Hash { + return ErrInvalidAdjacentHeaders + } + + now := System.Time() + if !isWithinTrustedPeriod(trustedHeader, trustingPeriod, now) { + return ErrHeaderNotWithinTrustedPeriod + } + + return nil + } +``` + +*Assumption*: In the following, we assume that *untrusted_h.Header.height > trusted_h.Header.height*. We will quickly discuss the other case in the next section. + +We consider the following set-up: + +- the light client communicates with one full node +- the light client locally stores all the headers that has passed basic verification and that are within light client trust period. In the pseudo code below we +write *Store.Add(header)* for this. If a header failed to verify, then +the full node we are talking to is faulty and we should disconnect from it and reinitialize with new peer. +- If `CanTrust` returns *error*, then the light client has seen a forged header or the trusted header has expired (it is outside its trusted period). + - In case of forged header, the full node is faulty so light client should disconnect and reinitialize with new peer. If the trusted header has expired, + we need to reinitialize light client with new trusted header (that is within its trusted period), but we don't necessarily need to disconnect from the full node + we are talking to (as we haven't observed full node misbehavior in this case). + +## Correctness of the Light Client Protocols + +### Definitions + +- `TRUSTED_PERIOD`: trusted period +- for realtime `t`, the predicate `correct(v,t)` is true if the validator `v` + follows the protocol until time `t` (we will see about recovery later). +- Validator fields. We will write a validator as a tuple `(v,p)` such that + - `v` is the identifier (i.e., validator address; we assume identifiers are unique in each validator set) + - `p` is its voting power +- For each header `h`, we write `trust(h) = true` if the light client trusts `h`. + +### Failure Model + +If a block `b` with a header `h` is generated at time `Time` (i.e. `h.Time = Time`), then a set of validators that +hold more than `2/3` of the voting power in `validators(h.NextValidatorsHash)` is correct until time +`h.Time + TRUSTED_PERIOD`. + +Formally, + +```latex +[ +\sum_{(v,p) \in validators(h.NextValidatorsHash) \wedge correct(v,h.Time + TRUSTED_PERIOD)} p > +\frac{2}{3} \sum_{(v,p) \in validators(h.NextValidatorsHash)} p +] +``` + + +The light client communicates with a full node and learns new headers. The goal is to locally decide whether to trust a header. Our implementation needs to ensure the following two properties: + +- *Light Client Completeness*: If a header `h` was correctly generated by an instance of Tendermint consensus (and its age is less than the trusted period), +then the light client should eventually set `trust(h)` to `true`. + +- *Light Client Accuracy*: If a header `h` was *not generated* by an instance of Tendermint consensus, then the light client should never set `trust(h)` to true. + +*Remark*: If in the course of the computation, the light client obtains certainty that some headers were forged by adversaries +(that is were not generated by an instance of Tendermint consensus), it may submit (a subset of) the headers it has seen as evidence of misbehavior. + +*Remark*: In Completeness we use "eventually", while in practice `trust(h)` should be set to true before `h.Time + TRUSTED_PERIOD`. If not, the header +cannot be trusted because it is too old. + +*Remark*: If a header `h` is marked with `trust(h)`, but it is too old at some point in time we denote with `now` (`h.Time + TRUSTED_PERIOD < now`), +then the light client should set `trust(h)` to `false` again at time `now`. + +*Assumption*: Initially, the light client has a header `inithead` that it trusts, that is, `inithead` was correctly generated by the Tendermint consensus. + +To reason about the correctness, we may prove the following invariant. + +*Verification Condition: light Client Invariant.* + For each light client `l` and each header `h`: +if `l` has set `trust(h) = true`, + then validators that are correct until time `h.Time + TRUSTED_PERIOD` have more than two thirds of the voting power in `validators(h.NextValidatorsHash)`. + +Formally, + +```latex +[ +\sum_{(v,p) \in validators(h.NextValidatorsHash) \wedge correct(v,h.Time + TRUSTED_PERIOD)} p > +\frac{2}{3} \sum_{(v,p) \in validators(h.NextValidatorsHash)} p +] +``` + +*Remark.* To prove the invariant, we will have to prove that the light client only trusts headers that were correctly generated by Tendermint consensus. +Then the formula above follows from the failure model. + +## Details + +**Observation 1.** If `h.Time + TRUSTED_PERIOD > now`, we trust the validator set `validators(h.NextValidatorsHash)`. + +When we say we trust `validators(h.NextValidatorsHash)` we do `not` trust that each individual validator in `validators(h.NextValidatorsHash)` +is correct, but we only trust the fact that less than `1/3` of them are faulty (more precisely, the faulty ones have less than `1/3` of the total voting power). + +*`VerifySingle` correctness arguments* + +Light Client Accuracy: + +- Assume by contradiction that `untrustedHeader` was not generated correctly and the light client sets trust to true because `verifySingle` returns without error. +- `trustedState` is trusted and sufficiently new +- by the Failure Model, less than `1/3` of the voting power held by faulty validators => at least one correct validator `v` has signed `untrustedHeader`. +- as `v` is correct up to now, it followed the Tendermint consensus protocol at least up to signing `untrustedHeader` => `untrustedHeader` was correctly generated. +We arrive at the required contradiction. + +Light Client Completeness: + +- The check is successful if sufficiently many validators of `trustedState` are still validators in the height `untrustedHeader.Height` and signed `untrustedHeader`. +- If `untrustedHeader.Height = trustedHeader.Height + 1`, and both headers were generated correctly, the test passes. + +*Verification Condition:* We may need an invariant stating that if `untrustedSignedHeader.Header.Height = trustedHeader.Height + 1` then +`signers(untrustedSignedHeader.Commit) \subseteq validators(trustedHeader.NextValidatorsHash)`. + +*Remark*: The variable `trustThreshold` can be used if the user believes that relying on one correct validator is not sufficient. +However, in case of (frequent) changes in the validator set, the higher the `trustThreshold` is chosen, the more unlikely it becomes that +`verifySingle` returns with an error for non-adjacent headers. + +- `VerifyBisection` correctness arguments (sketch)* + +Light Client Accuracy: + +- Assume by contradiction that the header at `untrustedHeight` obtained from the full node was not generated correctly and +the light client sets trust to true because `VerifyBisection` returns without an error. +- `VerifyBisection` returns without error only if all calls to `verifySingle` in the recursion return without error (return `nil`). +- Thus we have a sequence of headers that all satisfied the `verifySingle` +- again a contradiction + +light Client Completeness: + +This is only ensured if upon `Commit(pivot)` the light client is always provided with a correctly generated header. + +*Stalling* + +With `VerifyBisection`, a faulty full node could stall a light client by creating a long sequence of headers that are queried one-by-one by the light client and look OK, +before the light client eventually detects a problem. There are several ways to address this: + +- Each call to `Commit` could be issued to a different full node +- Instead of querying header by header, the light client tells a full node which header it trusts, and the height of the header it needs. The full node responds with +the header along with a proof consisting of intermediate headers that the light client can use to verify. Roughly, `VerifyBisection` would then be executed at the full node. +- We may set a timeout how long `VerifyBisection` may take. diff --git a/cometbft/v0.38/spec/p2p/Implementation-of-the-p2p-layer.mdx b/cometbft/v0.38/spec/p2p/Implementation-of-the-p2p-layer.mdx new file mode 100644 index 00000000..9011536b --- /dev/null +++ b/cometbft/v0.38/spec/p2p/Implementation-of-the-p2p-layer.mdx @@ -0,0 +1,43 @@ +--- +order: 1 +title: Implementation +--- + +# Implementation of the p2p layer + +This section documents the implementation of the peer-to-peer (p2p) +communication layer in CometBFT. + +The documentation was [produced](https://github.com/tendermint/tendermint/pull/9348) +using the `v0.34.*` releases +and the branch [`v0.34.x`](https://github.com/cometbft/cometbft/tree/v0.34.x) +of this repository as reference. +As there were no substancial changes in the p2p implementation, the +documentation also applies to the releases `v0.37.*` and `v0.38.*` [^v35]. + +[^v35]: The releases `v0.35.*` and `v0.36.*`, which included a major + refactoring of the p2p layer implementation, were [discontinued][v35postmorten]. + +[v35postmorten]: https://interchain-io.medium.com/discontinuing-tendermint-v0-35-a-postmortem-on-the-new-networking-layer-3696c811dabc + +## Contents + +The documentation follows the organization of the +[`p2p` package](https://github.com/cometbft/cometbft/tree/v0.34.x/p2p), +which implements the following abstractions: + +- [Transport](./transport.md): establishes secure and authenticated + connections with peers; +- [Switch](./switch.md): responsible for dialing peers and accepting + connections from peers, for managing established connections, and for + routing messages between the reactors and peers, + that is, between local and remote instances of the CometBFT protocols; +- [PEX Reactor](./pex.md): due to the several roles of this component, the + documentation is split in several parts: + - [Peer Exchange protocol](./pex-protocol.md): enables nodes to exchange peer addresses, thus implementing a peer discovery service; + - [Address Book](./addressbook.md): stores discovered peer addresses and + quality metrics associated to peers with which the node has interacted; + - [Peer Manager](./peer_manager.md): defines when and to which peers a node + should dial, in order to establish outbound connections; +- [Types](./types.md) and [Configuration](./configuration.md) provide a list of + existing types and configuration parameters used by the p2p package. diff --git a/cometbft/v0.38/spec/p2p/Peer-to-Peer.mdx b/cometbft/v0.38/spec/p2p/Peer-to-Peer.mdx new file mode 100644 index 00000000..29efd8ec --- /dev/null +++ b/cometbft/v0.38/spec/p2p/Peer-to-Peer.mdx @@ -0,0 +1,46 @@ +--- +order: 1 +parent: + title: P2P + order: 6 +--- + +# Peer-to-Peer Communication + +A CometBFT network is composed of multiple CometBFT instances, hereafter called +`nodes`, that interact by exchanging messages. + +The CometBFT protocols are designed under the assumption of a partially-connected network model. +This means that a node is not assumed to be directly connected to every other +node in the network. +Instead, each node is directly connected to only a subset of other nodes, +hereafter called its `peers`. + +The peer-to-peer (p2p) communication layer is then the component of CometBFT that: + +1. establishes connections between nodes in a CometBFT network +2. manages the communication between a node and the connected peers +3. intermediates the exchange of messages between peers in CometBFT protocols + +The specification the p2p layer is a work in progress, +tracked by [issue #19](https://github.com/cometbft/cometbft/issues/19). +The current content is organized as follows: + +- [`implementation`](./implementation/README.md): documents the current state + of the implementation of the p2p layer, covering the main components of the + `p2p` package. The documentation covers, in a fairly comprehensive way, + the items 1. and 2. from the list above. +- [`reactor-api`](./reactor-api/README.md): specifies the API offered by the + p2p layer to the protocol layer, through the `Reactor` abstraction. + This is a high-level specification (i.e., it should not be implementation-specific) + of the p2p layer API, covering item 3. from the list above. +- [`legacy-docs`](./legacy-docs/): We keep older documentation in + the `legacy-docs` directory, as overall, it contains useful information. + However, part of this content is redundant, + being more comprehensively covered in more recent documents, + and some implementation details might be outdated + (see [issue #981](https://github.com/cometbft/cometbft/issues/981)). + +In addition to this content, some unfinished, work in progress, and auxiliary +material can be found in the +[knowledge-base](https://github.com/cometbft/knowledge-base/tree/main/p2p) repository. diff --git a/cometbft/v0.38/spec/p2p/images/p2p-reactors.png b/cometbft/v0.38/spec/p2p/images/p2p-reactors.png new file mode 100644 index 00000000..5515976c Binary files /dev/null and b/cometbft/v0.38/spec/p2p/images/p2p-reactors.png differ diff --git a/cometbft/v0.38/spec/p2p/images/p2p_state.png b/cometbft/v0.38/spec/p2p/images/p2p_state.png new file mode 100644 index 00000000..c86d0168 Binary files /dev/null and b/cometbft/v0.38/spec/p2p/images/p2p_state.png differ diff --git a/cometbft/v0.38/spec/p2p/implementation/addressbook.md b/cometbft/v0.38/spec/p2p/implementation/addressbook.md new file mode 100644 index 00000000..26b95042 --- /dev/null +++ b/cometbft/v0.38/spec/p2p/implementation/addressbook.md @@ -0,0 +1,368 @@ +# Address Book + +The address book tracks information about peers, i.e., about other nodes in the network. + +The primary information stored in the address book are peer addresses. +A peer address is composed by a node ID and a network address; a network +address is composed by an IP address or a DNS name plus a port number. +The same node ID can be associated to multiple network addresses. + +There are two sources for the addresses stored in the address book. +The [Peer Exchange protocol](./pex-protocol.md) stores in the address book +the peer addresses it discovers, i.e., it learns from connected peers. +And the [Switch](./switch.md) registers the addresses of peers with which it +has interacted: to which it has dialed or from which it has accepted a +connection. + +The address book also records additional information about peers with which the +node has interacted, from which is possible to rank peers. +The Switch reports [connection attempts](#dial-attempts) to a peer address; too +much failed attempts indicate that a peer address is invalid. +Reactors, in they turn, report a peer as [good](#good-peers) when it behaves as +expected, or as a [bad peer](#bad-peers), when it misbehaves. + +There are two entities that retrieve peer addresses from the address book. +The [Peer Manager](./peer_manager.md) retrieves peer addresses to dial, so to +establish outbound connections. +This selection is random, but has a configurable bias towards peers that have +been marked as good peers. +The [Peer Exchange protocol](./pex-protocol.md) retrieves random samples of +addresses to offer (send) to peers. +This selection is also random but it includes, in particular for nodes that +operate in seed mode, some bias toward peers marked as good ones. + +## Buckets + +Peer addresses are stored in buckets. +There are buckets for new addresses and buckets for old addresses. +The buckets for new addresses store addresses of peers about which the node +does not have much information; the first address registered for a peer ID is +always stored in a bucket for new addresses. +The buckets for old addresses store addresses of peers with which the node has +interacted and that were reported as [good peers](#good-peers) by a reactor. +An old address therefore can be seen as an alias for a good address. + +> Note that new addresses does not mean bad addresses. +> The addresses of peers marked as [bad peers](#bad-peers) are removed from the +> buckets where they are stored, and temporarily kept in a table of banned peers. + +The number of buckets is fixed and there are more buckets for new addresses +(`256`) than buckets for old addresses (`64`), a ratio of 4:1. +Each bucket can store up to `64` addresses. +When a bucket becomes full, the peer address with the lowest ranking is removed +from the bucket. +The first choice is to remove bad addresses, with multiple failed attempts +associated. +In the absence of those, the *oldest* address in the bucket is removed, i.e., +the address with the oldest last attempt to dial. + +When a bucket for old addresses becomes full, the lowest-ranked peer address in +the bucket is moved to a bucket of new addresses. +When a bucket for new addresses becomes full, the lowest-ranked peer address in +the bucket is removed from the address book. +In other words, exceeding old or good addresses are downgraded to new +addresses, while exceeding new addresses are dropped. + +The bucket that stores an `address` is defined by the following two methods, +for new and old addresses: + +- `calcNewBucket(address, source) = hash(key + groupKey(source) + hash(key + groupKey(address) + groupKey(source)) % newBucketsPerGroup) % newBucketCount` +- `calcOldBucket(address) = hash(key + groupKey(address) + hash(key + address) % oldBucketsPerGroup) % oldBucketCount` + +The `key` is a fixed random 96-bit (8-byte) string. +The `groupKey` for an address is a string representing its network group. +The `source` of an address is the address of the peer from which we learn the +address.. +The first (internal) hash is reduced to an integer up to `newBucketsPerGroup = +32`, for new addresses, and `oldBucketsPerGroup = 4`, for old addresses. +The second (external) hash is reduced to bucket indexes, in the interval from 0 +to the number of new (`newBucketCount = 256`) or old (`oldBucketCount = 64`) buckets. + +Notice that new addresses with sources from the same network group are more +likely to end up in the same bucket, therefore to competing for it. +For old address, instead, two addresses are more likely to end up in the same +bucket when they belong to the same network group. + +## Adding addresses + +The `AddAddress` method adds the address of a peer to the address book. + +The added address is associated to a *source* address, which identifies the +node from which the peer address was learned. + +Addresses are added to the address book in the following situations: + +1. When a peer address is learned via PEX protocol, having the sender + of the PEX message as its source +2. When an inbound peer is added, in this case the peer itself is set as the + source of its own address +3. When the switch is instructed to dial addresses via the `DialPeersAsync` + method, in this case the node itself is set as the source + +If the added address contains a node ID that is not registered in the address +book, the address is added to a [bucket](#buckets) of new addresses. +Otherwise, the additional address for an existing node ID is **not added** to +the address book when: + +- The last address added with the same node ID is stored in an old bucket, so + it is considered a "good" address +- There are addresses associated to the same node ID stored in + `maxNewBucketsPerAddress = 4` distinct buckets +- Randomly, with a probability that increases exponentially with the number of + buckets in which there is an address with the same node ID. + So, a new address for a node ID which is already present in one bucket is + added with 50% of probability; if the node ID is present in two buckets, the + probability decreases to 25%; and if it is present in three buckets, the + probability is 12.5%. + +The new address is also added to the `addrLookup` table, which stores +`knownAddress` entries indexed by their node IDs. +If the new address is from an unknown peer, a new entry is added to the +`addrLookup` table; otherwise, the existing entry is updated with the new +address. +Entries of this table contain, among other fields, the list of buckets where +addresses of a peer are stored. +The `addrLookup` table is used by most of the address book methods (e.g., +`HasAddress`, `IsGood`, `MarkGood`, `MarkAttempt`), as it provides fast access +to addresses. + +### Errors + +- if the added address or the associated source address are nil +- if the added address is invalid +- if the added address is the local node's address +- if the added address ID is of a [banned](#bad-peers) peer +- if either the added address or the associated source address IDs are configured as private IDs +- if `routabilityStrict` is set and the address is not routable +- in case of failures computing the bucket for the new address (`calcNewBucket` method) +- if the added address instance, which is a new address, is configured as an + old address (sanity check of `addToNewBucket` method) + +## Need for Addresses + +The `NeedMoreAddrs` method verifies whether the address book needs more addresses. + +It is invoked by the PEX reactor to define whether to request peer addresses +to a new outbound peer or to a randomly selected connected peer. + +The address book needs more addresses when it has less than `1000` addresses +registered, counting all buckets for new and old addresses. + +## Pick address + +The `PickAddress` method returns an address stored in the address book, chosen +at random with a configurable bias towards new addresses. + +It is invoked by the Peer Manager to obtain a peer address to dial, as part of +its `ensurePeers` routine. +The bias starts from 10%, when the peer has no outbound peers, increasing by +10% for each outbound peer the node has, up to 90%, when the node has at least +8 outbound peers. + +The configured bias is a parameter that influences the probability of choosing +an address from a bucket of new addresses or from a bucket of old addresses. +A second parameter influencing this choice is the number of new and old +addresses stored in the address book. +In the absence of bias (i.e., if the configured bias is 50%), the probability +of picking a new address is given by the square root of the number of new +addresses divided by the sum of the square roots of the numbers of new and old +addresses. +By adding a bias toward new addresses (i.e., configured bias larger than 50%), +the portion on the sample occupied by the square root of the number of new +addresses increases, while the corresponding portion for old addresses decreases. +As a result, it becomes more likely to pick a new address at random from this sample. + +> The use of the square roots softens the impact of disproportional numbers of +> new and old addresses in the address book. This is actually the expected +> scenario, as there are 4 times more buckets for new addresses than buckets +> for old addresses. + +Once the type of address, new or old, is defined, a non-empty bucket of this +type is selected at random. +From the selected bucket, an address is chosen at random and returned. +If all buckets of the selected type are empty, no address is returned. + +## Random selection + +The `GetSelection` method returns a selection of addresses stored in the +address book, with no bias toward new or old addresses. + +It is invoked by the PEX protocol to obtain a list of peer addresses with two +purposes: + +- To send to a peer in a PEX response, in the case of outbound peers or of + nodes not operating in seed mode +- To crawl, in the case of nodes operating in seed mode, as part of every + interaction of the `crawlPeersRoutine` + +The selection is a random subset of the peer addresses stored in the +`addrLookup` table, which stores the last address added for each peer ID. +The target size of the selection is `23%` (`getSelectionPercent`) of the +number of addresses stored in the address book, but it should not be lower than +`32` (`minGetSelection`) --- if it is, all addresses in the book are returned +--- nor greater than `250` (`maxGetSelection`). + +> The random selection is produced by: +> +> - Retrieving all entries of the `addrLookup` map, which by definition are +> returned in random order. +> - Randomly shuffling the retrieved list, using the Fisher-Yates algorithm + +## Random selection with bias + +The `GetSelectionWithBias` method returns a selection of addresses stored in +the address book, with bias toward new addresses. + +It is invoked by the PEX protocol to obtain a list of peer addresses to be sent +to a peer in a PEX response. +This method is only invoked by seed nodes, when replying to a PEX request +received from an inbound peer (i.e., a peer that dialed the seed node). +The bias used in this scenario is hard-coded to 30%, meaning that 70% of +the returned addresses are expected to be old addresses. + +The number of addresses that compose the selection is computed in the same way +as for the non-biased random selection. +The bias toward new addresses is implemented by requiring that the configured +bias, interpreted as a percentage, of the select addresses come from buckets of +new addresses, while the remaining come from buckets of old addresses. +Since the number of old addresses is typically lower than the number of new +addresses, it is possible that the address book does not have enough old +addresses to include in the selection. +In this case, additional new addresses are included in the selection. +Thus, the configured bias, in practice, is towards old addresses, not towards +new addresses. + +To randomly select addresses of a type, the address book considers all +addresses present in every bucket of that type. +This list of all addresses of a type is randomly shuffled, and the requested +number of addresses are retrieved from the tail of this list. +The returned selection contains, at its beginning, a random selection of new +addresses in random order, followed by a random selection of old addresses, in +random order. + +## Dial Attempts + +The `MarkAttempt` method records a failed attempt to connect to an address. + +It is invoked by the Peer Manager when it fails dialing a peer, but the failure +is not in the authentication step (`ErrSwitchAuthenticationFailure` error). +In case of authentication errors, the peer is instead marked as a [bad peer](#bad-peers). + +The failed connection attempt is recorded in the address registered for the +peer's ID in the `addrLookup` table, which is the last address added with that ID. +The known address' counter of failed `Attempts` is increased and the failure +time is registered in `LastAttempt`. + +The possible effect of recording multiple failed connect attempts to a peer is +to turn its address into a *bad* address (do not confuse with banned addresses). +A known address becomes bad if it is stored in buckets of new addresses, and +when connection attempts: + +- Have not been made over a week, i.e., `LastAttempt` is older than a week +- Have failed 3 times and never succeeded, i.e., `LastSucess` field is unset +- Have failed 10 times in the last week, i.e., `LastSucess` is older than a week + +Addresses marked as *bad* are the first candidates to be removed from a bucket of +new addresses when the bucket becomes full. + +> Note that failed connection attempts are reported for a peer address, but in +> fact the address book records them for a peer. +> +> More precisely, failed connection attempts are recorded in the entry of the +> `addrLookup` table with reported peer ID, which contains the last address +> added for that node ID, which is not necessarily the reported peer address. + +## Good peers + +The `MarkGood` method marks a peer ID as good. + +It is invoked by the consensus reactor, via switch, when the number of useful +messages received from a peer is a multiple of `10000`. +Vote and block part messages are considered for this number, they must be valid +and not be duplicated messages to be considered useful. + +> The `SwitchReporter` type of `behaviour` package also invokes the `MarkGood` +> method when a "reason" associated with consensus votes and block parts is +> reported. +> No reactor, however, currently provides these "reasons" to the `SwitchReporter`. + +The effect of this action is that the address registered for the peer's ID in the +`addrLookup` table, which is the last address added with that ID, is marked as +good and moved to a bucket of old addresses. +An address marked as good has its failed to connect counter and timestamp reset. +If the destination bucket of old addresses is full, the oldest address in the +bucket is moved (downgraded) to a bucket of new addresses. + +Moving the peer address to a bucket of old addresses has the effect of +upgrading, or increasing the ranking of a peer in the address book. + +## Bad peers + +The `MarkBad` method marks a peer as bad and bans it for a period of time. + +This method is only invoked within the PEX reactor, with a banning time of 24 +hours, for the following reasons: + +- A peer misbehaves in the [PEX protocol](./pex-protocol.md#misbehavior) +- When the `maxAttemptsToDial` limit (`16`) is reached for a peer +- If an `ErrSwitchAuthenticationFailure` error is returned when dialing a peer + +The effect of this action is that the address registered for the peer's ID in the +`addrLookup` table, which is the last address added with that ID, is banned for +a period of time. +The banned peer is removed from the `addrLookup` table and from all buckets +where its addresses are stored. + +The information about banned peers, however, is not discarded. +It is maintained in the `badPeers` map, indexed by peer ID. +This allows, in particular, addresses of banned peers to be +[reinstated](#reinstating-addresses), i.e., to be added +back to the address book, when their ban period expires. + +## Reinstating addresses + +The `ReinstateBadPeers` method attempts to re-add banned addresses to the address book. + +It is invoked by the PEX reactor when dialing new peers. +This action is taken before requesting additional addresses to peers, +in the case that the node needs more peer addresses. + +The set of banned peer addresses is retrieved from the `badPeers` map. +Addresses that are not any longer banned, i.e., whose banned period has expired, +are added back to the address book as new addresses, while the corresponding +node IDs are removed from the `badPeers` map. + +## Removing addresses + +The `RemoveAddress` method removes an address from the address book. + +It is invoked by the switch when it dials a peer or accepts a connection from a +peer that ends up being the node itself (`IsSelf` error). +In both cases, the address dialed or accepted is also added to the address book +as a local address, via the `AddOurAddress` method. + +The same logic is also internally used by the address book for removing +addresses of a peer that is [marked as a bad peer](#bad-peers). + +The entry registered with the peer ID of the address in the `addrLookup` table, +which is the last address added with that ID, is removed from all buckets where +it is stored and from the `addrLookup` table. + +> FIXME: is it possible that addresses with the same ID as the removed address, +> but with distinct network addresses, are kept in buckets of the address book? +> While they will not be accessible anymore, as there is no reference to them +> in the `addrLookup`, they will still be there. + +## Persistence + +The `loadFromFile` method, called when the address book is started, reads +address book entries from a file, passed to the address book constructor. +The file, at this point, does not need to exist. + +The `saveRoutine` is started when the address book is started. +It saves the address book to the configured file every `dumpAddressInterval`, +hard-coded to 2 minutes. +It is also possible to save the content of the address book using the `Save` +method. +Saving the address book content to a file acquires the address book lock, also +employed by all other public methods. diff --git a/cometbft/v0.38/spec/p2p/implementation/configuration.md b/cometbft/v0.38/spec/p2p/implementation/configuration.md new file mode 100644 index 00000000..9f172c22 --- /dev/null +++ b/cometbft/v0.38/spec/p2p/implementation/configuration.md @@ -0,0 +1,49 @@ +# CometBFT p2p configuration + +This document contains configurable parameters a node operator can use to tune the p2p behaviour. + +| Parameter| Default| Description | +| --- | --- | ---| +| ListenAddress | "tcp://0.0.0.0:26656" | Address to listen for incoming connections (0.0.0.0:0 means any interface, any port) | +| ExternalAddress | "" | Address to advertise to peers for them to dial | +| [Seeds](./pex-protocol.md#seed-nodes) | empty | Comma separated list of seed nodes to connect to (ID@host:port )| +| [Persistent peers](./peer_manager.md#persistent-peers) | empty | Comma separated list of nodes to keep persistent connections to (ID@host:port ) | +| [AddrBook](./addressbook.md) | defaultAddrBookPath | Path do address book | +| AddrBookStrict | true | Set true for strict address routability rules and false for private or local networks | +| [MaxNumInboundPeers](./switch.md#accepting-peers) | 40 | Maximum number of inbound peers | +| [MaxNumOutboundPeers](./peer_manager.md#ensure-peers) | 10 | Maximum number of outbound peers to connect to, excluding persistent peers | +| [UnconditionalPeers](./switch.md#accepting-peers) | empty | These are IDs of the peers which are allowed to be (re)connected as both inbound or outbound regardless of whether the node reached `max_num_inbound_peers` or `max_num_outbound_peers` or not. | +| PersistentPeersMaxDialPeriod| 0 * time.Second | Maximum pause when redialing a persistent peer (if zero, exponential backoff is used) | +| FlushThrottleTimeout |100 * time.Millisecond| Time to wait before flushing messages out on the connection | +| MaxPacketMsgPayloadSize | 1024 | Maximum size of a message packet payload, in bytes | +| SendRate | 5120000 (5 mB/s) | Rate at which packets can be sent, in bytes/second | +| RecvRate | 5120000 (5 mB/s) | Rate at which packets can be received, in bytes/second| +| [PexReactor](./pex.md) | true | Set true to enable the peer-exchange reactor | +| SeedMode | false | Seed mode, in which node constantly crawls the network and looks for. Does not work if the peer-exchange reactor is disabled. | +| PrivatePeerIDs | empty | Comma separated list of peer IDsthat we do not add to the address book or gossip to other peers. They stay private to us. | +| AllowDuplicateIP | false | Toggle to disable guard against peers connecting from the same ip.| +| [HandshakeTimeout](./transport.md#connection-upgrade) | 20 * time.Second | Timeout for handshake completion between peers | +| [DialTimeout](./switch.md#dialing-peers) | 3 * time.Second | Timeout for dialing a peer | + + +These parameters can be set using the `$CMTHOME/config/config.toml` file. A subset of them can also be changed via command line using the following command line flags: + +| Parameter | Flag | Example | +| --- | --- | --- | +| Listen address| `p2p.laddr` | "tcp://0.0.0.0:26656" | +| Seed nodes | `p2p.seeds` | `--p2p.seeds “id100000000000000000000000000000000@1.2.3.4:26656,id200000000000000000000000000000000@2.3.4.5:4444”` | +| Persistent peers | `p2p.persistent_peers` | `--p2p.persistent_peers “id100000000000000000000000000000000@1.2.3.4:26656,id200000000000000000000000000000000@2.3.4.5:26656”` | +| Unconditional peers | `p2p.unconditional_peer_ids` | `--p2p.unconditional_peer_ids “id100000000000000000000000000000000,id200000000000000000000000000000000”` | +| PexReactor | `p2p.pex` | `--p2p.pex` | +| Seed mode | `p2p.seed_mode` | `--p2p.seed_mode` | +| Private peer ids | `p2p.private_peer_ids` | `--p2p.private_peer_ids “id100000000000000000000000000000000,id200000000000000000000000000000000”` | + + **Note on persistent peers** + + If `persistent_peers_max_dial_period` is set greater than zero, the +pause between each dial to each persistent peer will not exceed `persistent_peers_max_dial_period` +during exponential backoff and we keep trying again without giving up. + +If `seeds` and `persistent_peers` intersect, +the user will be warned that seeds may auto-close connections +and that the node may not be able to keep the connection persistent. diff --git a/cometbft/v0.38/spec/p2p/implementation/peer_manager.md b/cometbft/v0.38/spec/p2p/implementation/peer_manager.md new file mode 100644 index 00000000..5dfc14b2 --- /dev/null +++ b/cometbft/v0.38/spec/p2p/implementation/peer_manager.md @@ -0,0 +1,140 @@ +# Peer Manager + +The peer manager is responsible for establishing connections with peers. +It defines when a node should dial peers and which peers it should dial. +The peer manager is not an implementation abstraction of the p2p layer, +but a role that is played by the [PEX reactor](./pex.md). + +## Outbound peers + +The `ensurePeersRoutine` is a persistent routine intended to ensure that a node +is connected to `MaxNumOutboundPeers` outbound peers. +This routine is continuously executed by regular nodes, i.e. nodes not +operating in seed mode, as part of the PEX reactor implementation. + +The logic defining when the node should dial peers, for selecting peers to dial +and for actually dialing them is implemented in the `ensurePeers` method. +This method is periodically invoked -- every `ensurePeersPeriod`, with default +value to 30 seconds -- by the `ensurePeersRoutine`. + +A node is expected to dial peers whenever the number of outbound peers is lower +than the configured `MaxNumOutboundPeers` parameter. +The current number of outbound peers is retrieved from the switch, using the +`NumPeers` method, which also reports the number of nodes to which the switch +is currently dialing. +If the number of outbound peers plus the number of dialing routines equals to +`MaxNumOutboundPeers`, nothing is done. +Otherwise, the `ensurePeers` method will attempt to dial node addresses in +order to reach the target number of outbound peers. + +Once defined that the node needs additional outbound peers, the node queries +the address book for candidate addresses. +This is done using the [`PickAddress`](./addressbook.md#pick-address) method, +which returns an address selected at random on the address book, with some bias +towards new or old addresses. +When the node has up to 3 outbound peers, the adopted bias is towards old +addresses, i.e., addresses of peers that are believed to be "good". +When the node has from 5 outbound peers, the adopted bias is towards new +addresses, i.e., addresses of peers about which the node has not yet collected +much information. +So, the more outbound peers a node has, the less conservative it will be when +selecting new peers. + +The selected peer addresses are then dialed in parallel, by starting a dialing +routine per peer address. +Dialing a peer address can fail for multiple reasons. +The node might have attempted to dial the peer too many times. +In this case, the peer address is marked as bad and removed from the address book. +The node might have attempted and failed to dial the peer recently +and the exponential `backoffDuration` has not yet passed. +Or the current connection attempt might fail, which is registered in the address book. +None of these errors are explicitly handled by the `ensurePeers` method, which +also does not wait until the connections are established. + +The third step of the `ensurePeers` method is to ensure that the address book +has enough addresses. +This is done, first, by [reinstating banned peers](./addressbook.md#Reinstating-addresses) +whose ban period has expired. +Then, the node randomly selects a connected peer, which can be either an +inbound or outbound peer, to [requests addresses](./pex-protocol.md#Requesting-Addresses) +using the PEX protocol. +Last, and this action is only performed if the node could not retrieve any new +address to dial from the address book, the node dials the configured seed nodes +in order to establish a connection to at least one of them. + +### Fast dialing + +As above described, seed nodes are actually the last source of peer addresses +for regular nodes. +They are contacted by a node when, after an invocation of the `ensurePeers` +method, no suitable peer address to dial is retrieved from the address book +(e.g., because it is empty). + +Once a connection with a seed node is established, the node immediately +[sends a PEX request](./pex-protocol.md#Requesting-Addresses) to it, as it is +added as an outbound peer. +When the corresponding PEX response is received, the addresses provided by the +seed node are added to the address book. +As a result, in the next invocation of the `ensurePeers` method, the node +should be able to dial some of the peer addresses provided by the seed node. + +However, as observed in this [issue](https://github.com/tendermint/tendermint/issues/2093), +it can take some time, up to `ensurePeersPeriod` or 30 seconds, from when the +node receives new peer addresses and when it dials the received addresses. +To avoid this delay, which can be particularly relevant when the node has no +peers, a node immediately attempts to dial peer addresses when they are +received from a peer that is locally configured as a seed node. + +> This was implemented in a rough way, leading to inconsistencies described in +> this [issue](https://github.com/cometbft/cometbft/issues/486), +> fixed by this [PR](https://github.com/cometbft/cometbft/pull/3360). + +### First round + +When the PEX reactor is started, the `ensurePeersRoutine` is created and it +runs thorough the operation of a node, periodically invoking the `ensurePeers` +method. +However, if when the persistent routine is started the node already has some +peers, either inbound or outbound peers, or is dialing some addresses, the +first invocation of `ensurePeers` is delayed by a random amount of time from 0 +to `ensurePeersPeriod`. + +### Persistent peers + +The node configuration can contain a list of *persistent peers*. +Those peers have preferential treatment compared to regular peers and the node +is always trying to connect to them. +Moreover, these peers are not removed from the address book in the case of +multiple failed dial attempts. + +On startup, the node immediately tries to dial the configured persistent peers +by calling the switch's [`DialPeersAsync`](./switch.md#manual-operation) method. +This is not done in the p2p package, but it is part of the procedure to set up a node. + +> TODO: the handling of persistent peers should be described in more detail. + +### Life cycle + +The picture below is a first attempt of illustrating the life cycle of an outbound peer: + + + +A peer can be in the following states: + +- Candidate peers: peer addresses stored in the address boook, that can be + retrieved via the [`PickAddress`](./addressbook.md#pick-address) method +- [Dialing](./switch.md#dialing-peers): peer addresses that are currently being + dialed. This state exists to ensure that a single dialing routine exist per peer. +- [Reconnecting](./switch.md#reconnect-to-peer): persistent peers to which a node + is currently reconnecting, as a previous connection attempt has failed. +- Connected peers: peers that a node has successfully dialed, added as outbound peers. +- [Bad peers](./addressbook.md#bad-peers): peers marked as bad in the address + book due to exhibited [misbehavior](./pex-protocol.md#misbehavior). + Peers can be reinstated after being marked as bad. + +## Pending of documentation + +The `dialSeeds` method of the PEX reactor. + +The `dialPeer` method of the PEX reactor. +This includes `dialAttemptsInfo`, `maxBackoffDurationForPeer` methods. diff --git a/cometbft/v0.38/spec/p2p/implementation/pex-protocol.md b/cometbft/v0.38/spec/p2p/implementation/pex-protocol.md new file mode 100644 index 00000000..760a56bd --- /dev/null +++ b/cometbft/v0.38/spec/p2p/implementation/pex-protocol.md @@ -0,0 +1,240 @@ +# Peer Exchange Protocol + +The Peer Exchange (PEX) protocol enables nodes to exchange peer addresses, thus +implementing a peer discovery mechanism. + +The PEX protocol uses two messages: + +- `PexRequest`: sent by a node to [request](#requesting-addresses) peer + addresses to a peer +- `PexAddrs`: a list of peer addresses [provided](#providing-addresses) to a + peer as response to a `PexRequest` message + +While all nodes, with few exceptions, participate on the PEX protocol, +a subset of nodes, configured as [seed nodes](#seed-nodes) have a particular +role in the protocol. +They crawl the network, connecting to random peers, in order to learn as many +peer addresses as possible to provide to other nodes. + +## Requesting Addresses + +A node requests peer addresses by sending a `PexRequest` message to a peer. + +For regular nodes, not operating in seed mode, a PEX request is sent when +the node *needs* peers addresses, a condition checked: + +1. When an *outbound* peer is added, causing the node to request addresses from + the new peer +2. Periodically, by the `ensurePeersRoutine`, causing the node to request peer + addresses to a randomly selected peer + +A node needs more peer addresses when its addresses book has +[less than 1000 records](./addressbook.md#need-for-addresses). +It is thus reasonable to assume that the common case is that a peer needs more +peer addresses, so that PEX requests are sent whenever the above two situations happen. + +A PEX request is sent when a new *outbound* peer is added. +The same does not happen with new inbound peers because the implementation +considers outbound peers, that the node has chosen for dialing, more +trustworthy than inbound peers, that the node has accepted. +Moreover, when a node is short of peer addresses, it dials the configured seed nodes; +since they are added as outbound peers, the node can immediately request peer addresses. + +The `ensurePeersRoutine` periodically checks, by default every 30 seconds (`ensurePeersPeriod`), +whether the node has enough outbound peers. +If it does not have, the node tries dialing some peer addresses stored in the address book. +As part of this procedure, the node selects a peer at random, +from the set of connected peers retrieved from the switch, +and sends a PEX request to the selected peer. + +Sending a PEX request to a peer is implemented by the `RequestAddrs` method of +the PEX reactor. + +### Responses + +After a PEX request is sent to a peer, the node expects to receive, +as a response, a `PexAddrs` message from the peer. +This message encodes a list of peer addresses that are +[added to address book](./addressbook.md#adding-addresses), +having the peer from which the PEX response was received as their source. + +Received PEX responses are handled by the `ReceiveAddrs` method of the PEX reactor. +In the case of a PEX response received from a peer which is configured as +a seed node, the PEX reactor attempts immediately to dial the provided peer +addresses, as detailed [here](./peer_manager.md#fast-dialing). + +### Misbehavior + +Sending multiple PEX requests to a peer, before receiving a reply from it, +is considered a misbehavior. +To prevent it, the node maintains a `requestsSent` set of outstanding +requests, indexed by destination peers. +While a peer ID is present in the `requestsSent` set, the node does not send +further PEX requests to that peer. +A peer ID is removed from the `requestsSent` set when a PEX response is +received from it. + +Sending a PEX response to a peer that has not requested peer addresses +is also considered a misbehavior. +So, if a PEX response is received from a peer that is not registered in +the `requestsSent` set, a `ErrUnsolicitedList` error is produced. +This leads the peer to be disconnected and [marked as a bad peer](./addressbook.md#bad-peers). + +## Providing Addresses + +When a node receives a `PexRequest` message from a peer, +it replies with a `PexAddrs` message. + +This message encodes a [random selection of peer addresses](./addressbook.md#random-selection) +retrieved from the address book. + +Sending a PEX response to a peer is implemented by the `SendAddrs` method of +the PEX reactor. + +### Misbehavior + +Requesting peer addresses too often is considered a misbehavior. +Since node are expected to send PEX requests every `ensurePeersPeriod`, +the minimum accepted interval between requests from the same peer is set +to `ensurePeersPeriod / 3`, 10 seconds by default. + +The `receiveRequest` method is responsible for verifying this condition. +The node keeps a `lastReceivedRequests` map with the time of the last PEX +request received from every peer. +If the interval between successive requests is less than the minimum accepted +one, the peer is disconnected and [marked as a bad peer](./addressbook.md#bad-peers). +An exception is made for the first two PEX requests received from a peer. + +> The probably reason is that, when a new peer is added, the two conditions for +> a node to request peer addresses can be triggered with an interval lower than +> the minimum accepted interval. +> Since this is a legit behavior, it should not be punished. + +## Seed nodes + +A seed node is a node configured to operate in `SeedMode`. + +### Crawling peers + +Seed nodes crawl the network, connecting to random peers and sending PEX +requests to them, in order to learn as many peer addresses as possible. +More specifically, a node operating in seed mode sends PEX requests in two cases: + +1. When an outbound peer is added, and the seed node needs more peer addresses, + it requests peer addresses to the new peer +2. Periodically, the `crawlPeersRoutine` sends PEX requests to a random set of + peers, whose addresses are registered in the Address Book + +The first case also applies for nodes not operating in seed mode. +The second case replaces the second for regular nodes, as seed nodes do not +run the `ensurePeersRoutine`, as regular nodes, +but run the `crawlPeersRoutine`, which is not run by regular nodes. + +The `crawlPeersRoutine` periodically, every 30 seconds (`crawlPeerPeriod`), +starts a new peer discovery round. +First, the seed node retrieves a random selection of peer addresses from its +Address Book. +This selection is produced in the same way as in the random selection of peer +addresses that are [provided](#providing-addresses) to a requesting peer. +Peers that the seed node has crawled recently, +less than 2 minutes ago (`minTimeBetweenCrawls`), are removed from this selection. +The remaining peer addresses are registered in the `crawlPeerInfos` table. + +The seed node is not necessarily connected to the peer whose address is +selected for each round of crawling. +So, the seed node dials the selected peer addresses. +This is performed in foreground, one peer at a time. +As a result, a round of crawling can take a substantial amount of time. +For each selected peer it succeeds dialing to, this include already connected +peers, the seed node sends a PEX request. + +Dialing a selected peer address can fail for multiple reasons. +The seed node might have attempted to dial the peer too many times. +In this case, the peer address is marked as [bad in the address book](./addressbook.md#bad-peers). +The seed node might have attempted to dial the peer recently, without success, +and the exponential `backoffDuration` has not yet passed. +Or the current connection attempt might fail, which is registered in the address book. + +Failures to dial to a peer address produce an information that is important for +a seed node. +They indicate that a peer is unreachable, or is not operating correctly, and +therefore its address should not be provided to other nodes. +This occurs when, due to multiple failed connection attempts or authentication +failures, the peer address ends up being removed from the address book. +As a result, the periodically crawling of selected peers not only enables the +discovery of new peers, but also allows the seed node to stop providing +addresses of bad peers. + +### Offering addresses + +Nodes operating in seed mode handle PEX requests differently than regular +nodes, whose operation is described [here](#providing-addresses). + +This distinction exists because nodes dial a seed node with the main, if not +exclusive goal of retrieving peer addresses. +In other words, nodes do not dial a seed node because they intend to have it as +a peer in the multiple CometBFT protocols, but because they believe that a +seed node is a good source of addresses of nodes to which they can establish +connections and interact in the multiple CometBFT protocols. + +So, when a seed node receives a `PexRequest` message from an inbound peer, +it sends a `PexAddrs` message, containing a selection of peer +addresses, back to the peer and *disconnects* from it. +Seed nodes therefore treat inbound connections from peers as a short-term +connections, exclusively intended to retrieve peer addresses. +Once the requested peer addresses are sent, the connection with the peer is closed. + +Moreover, the selection of peer addresses provided to inbound peers by a seed +node, although still essentially random, has a [bias toward old +addresses](./addressbook.md#random-selection-with-bias). +The selection bias is defined by `biasToSelectNewPeers`, hard-coded to `30%`, +meaning that `70%` of the peer addresses provided by a seed node are expected +to be old addresses. +Although this nomenclature is not clear, *old* addresses are the addresses that +survived the most in the address book, that is, are addresses that the seed +node believes being from *good* peers (more details [here](./addressbook.md#good-peers)). + +Another distinction is on the handling of potential [misbehavior](#misbehavior-1) +of peers requesting addresses. +A seed node does not enforce, a priori, a minimal interval between PEX requests +from inbound peers. +Instead, it does not reply to more than one PEX request per peer inbound +connection, and, as above mentioned, it disconnects from incoming peers after +responding to them. +If the same peer dials again to the seed node and requests peer addresses, the +seed node will reply to this peer like it was the first time it has requested +peer addresses. + +> This is more an implementation restriction than a desired behavior. +> The `lastReceivedRequests` map stores the last time a PEX request was +> received from a peer, and the entry relative to a peer is removed from this +> map when the peer is disconnected. +> +> It is debatable whether this approach indeed prevents abuse against seed nodes. + +### Disconnecting from peers + +Seed nodes treat connections with peers as short-term connections, which are +mainly, if not exclusively, intended to exchange peer addresses. + +In the case of inbound peers, that have dialed the seed node, the intent of the +connection is achieved once a PEX response is sent to the peer. +The seed node thus disconnects from an inbound peer after sending a `PexAddrs` +message to it. + +In the case of outbound peers, which the seed node has dialed for crawling peer +addresses, the intent of the connection is essentially achieved when a PEX +response is received from the peer. +The seed node, however, does not disconnect from a peer after receiving a +selection of peer addresses from it. +As a result, after some rounds of crawling, a seed node will have established +connections to a substantial amount of peers. + +To couple with the existence of multiple connections with peers that have no +longer purpose for the seed node, the `crawlPeersRoutine` also invokes, after +each round of crawling, the `attemptDisconnects` method. +This method retrieves the list of connected peers from the switch, and +disconnects from peers that are not persistent peers, and with which a +connection is established for more than `SeedDisconnectWaitPeriod`. +This period is a configuration parameter, set to 28 hours when the PEX reactor +is created by the default node constructor. diff --git a/cometbft/v0.38/spec/p2p/implementation/pex.md b/cometbft/v0.38/spec/p2p/implementation/pex.md new file mode 100644 index 00000000..8243eaa5 --- /dev/null +++ b/cometbft/v0.38/spec/p2p/implementation/pex.md @@ -0,0 +1,111 @@ +# PEX Reactor + +The PEX reactor is one of the reactors running in a CometBFT node. + +Its implementation is located in the `p2p/pex` package, and it is considered +part of the implementation of the p2p layer. + +This document overviews the implementation of the PEX reactor, describing how +the methods from the `Reactor` interface are implemented. + +The actual operation of the PEX reactor is presented in documents describing +the roles played by the PEX reactor in the p2p layer: + +- [Address Book](./addressbook.md): stores known peer addresses and information + about peers to which the node is connected or has attempted to connect +- [Peer Manager](./peer_manager.md): manages connections established with peers, + defining when a node should dial peers and which peers it should dial +- [Peer Exchange protocol](./pex-protocol.md): enables nodes to exchange peer + addresses, thus implementing a peer discovery service + +## OnStart + +The `OnStart` method implements `BaseService` and starts the PEX reactor. + +The [address book](./addressbook.md), which is a `Service` is started. +This loads the address book content from disk, +and starts a routine that periodically persists the address book content to disk. + +The PEX reactor is configured with the addresses of a number of seed nodes, +the `Seeds` parameter of the `ReactorConfig`. +The addresses of seed nodes are parsed into `NetAddress` instances and resolved +into IP addresses, which is implemented by the `checkSeeds` method. +Valid seed node addresses are stored in the `seedAddrs` field, +and are used by the `dialSeeds` method to contact the configured seed nodes. + +The last action is to start one of the following persistent routines, based on +the `SeedMode` configuration parameter: + +- Regular nodes run the `ensurePeersRoutine` to check whether the node has + enough outbound peers, dialing peers when necessary +- Seed nodes run the `crawlPeersRoutine` to periodically start a new round + of [crawling](./pex-protocol.md#Crawling-peers) to discover as many peer + addresses as possible + +### Errors + +Errors encountered when loading the address book from disk are returned, +and prevent the reactor from being started. +An exception is made for the `service.ErrAlreadyStarted` error, which is ignored. + +Errors encountered when parsing the configured addresses of seed nodes +are returned and cause the reactor startup to fail. +An exception is made for DNS resolution `ErrNetAddressLookup` errors, +which are not deemed fatal and are only logged as invalid addresses. + +If none of the configured seed node addresses is valid, and the loaded address +book is empty, the reactor is not started and an error is returned. + +## OnStop + +The `OnStop` method implements `BaseService` and stops the PEX reactor. + +The address book routine that periodically saves its content to disk is stopped. + +## GetChannels + +The `GetChannels` method, from the `Reactor` interface, returns the descriptor +of the channel used by the PEX protocol. + +The channel ID is `PexChannel` (0), with priority `1`, send queue capacity of +`10`, and maximum message size of `64000` bytes. + +## AddPeer + +The `AddPeer` method, from the `Reactor` interface, +adds a new peer to the PEX protocol. + +If the new peer is an **inbound peer**, i.e., if the peer has dialed the node, +the peer's address is [added to the address book](./addressbook.md#adding-addresses). +Since the peer was authenticated when establishing a secret connection with it, +the source of the peer address is trusted, and its source is set by the peer itself. +In the case of an outbound peer, the node should already have its address in +the address book, as the switch has dialed the peer. + +If the peer is an **outbound peer**, i.e., if the node has dialed the peer, +and the PEX protocol needs more addresses, +the node [sends a PEX request](./pex-protocol.md#Requesting-Addresses) to the peer. +The same is not done when inbound peers are added because they are deemed least +trustworthy than outbound peers. + +## RemovePeer + +The `RemovePeer` method, from the `Reactor` interface, +removes a peer from the PEX protocol. + +The peer's ID is removed from the tables tracking PEX requests +[sent](./pex-protocol.md#misbehavior) but not yet replied +and PEX requests [received](./pex-protocol.md#misbehavior-1). + +## Receive + +The `Receive` method, from the `Reactor` interface, +handles a message received by the PEX protocol. + +A node receives two type of messages as part of the PEX protocol: + +- `PexRequest`: a request for addresses received from a peer, handled as + described [here](./pex-protocol.md#providing-addresses) +- `PexAddrs`: a list of addresses received from a peer, as a reponse to a PEX + request sent by the node, as described [here](./pex-protocol.md#responses) + diff --git a/cometbft/v0.38/spec/p2p/implementation/switch.md b/cometbft/v0.38/spec/p2p/implementation/switch.md new file mode 100644 index 00000000..4497fef9 --- /dev/null +++ b/cometbft/v0.38/spec/p2p/implementation/switch.md @@ -0,0 +1,238 @@ +# Switch + +The switch is a core component of the p2p layer. +It manages the procedures for [dialing peers](#dialing-peers) and +[accepting](#accepting-peers) connections from peers, which are actually +implemented by the [transport](./transport.md). +It also manages the reactors, i.e., protocols implemented by the node that +interact with its peers. +Once a connection with a peer is established, the peer is [added](#add-peer) to +the switch and all registered reactors. +Reactors may also instruct the switch to [stop a peer](#stop-peer), namely +disconnect from it. +The switch, in this case, makes sure that the peer is removed from all +registered reactors. + +## Dialing peers + +Dialing a peer is implemented by the `DialPeerWithAddress` method. + +This method is invoked by the [peer manager](./peer_manager.md#ensure-peers) +to dial a peer address and establish a connection with an outbound peer. + +The switch keeps a single dialing routine per peer ID. +This is ensured by keeping a synchronized map `dialing` with the IDs of peers +to which the peer is dialing. +A peer ID is added to `dialing` when the `DialPeerWithAddress` method is called +for that peer, and it is removed when the method returns for whatever reason. +The method returns immediately when invoked for a peer which ID is already in +the `dialing` structure. + +The actual dialing is implemented by the [`Dial`](./transport.md#dial) method +of the transport configured for the switch, in the `addOutboundPeerWithConfig` +method. +If the transport succeeds establishing a connection, the returned `Peer` is +added to the switch using the [`addPeer`](#add-peer) method. +This operation can fail, returning an error. In this case, the switch invokes +the transport's [`Cleanup`](./transport.md#cleanup) method to clean any resources +associated with the peer. + +If the transport fails to establish a connection with the peer that is configured +as a persistent peer, the switch spawns a routine to [reconnect to the peer](#reconnect-to-peer). +If the peer is already in the `reconnecting` state, the spawned routine has no +effect and returns immediately. +This is in fact a likely scenario, as the `reconnectToPeer` routine relies on +this same `DialPeerWithAddress` method for dialing peers. + +### Manual operation + +The `DialPeersAsync` method receives a list of peer addresses (strings) +and dials all of them in parallel. +It is invoked in two situations: + +- In the [setup](https://github.com/cometbft/cometbft/blob/v0.34.x/node/node.go#L987) +of a node, to establish connections with every configured persistent peer +- In the RPC package, to implement two unsafe RPC commands, not used in production: + [`DialSeeds`](https://github.com/cometbft/cometbft/blob/v0.34.x/rpc/core/net.go#L47) and + [`DialPeers`](https://github.com/cometbft/cometbft/blob/v0.34.x/rpc/core/net.go#L87) + +The received list of peer addresses to dial is parsed into `NetAddress` instances. +In case of parsing errors, the method returns. An exception is made for +DNS resolution `ErrNetAddressLookup` errors, which do not interrupt the procedure. + +As the peer addresses provided to this method are typically not known by the node, +contrarily to the addressed dialed using the `DialPeerWithAddress` method, +they are added to the node's address book, which is persisted to disk. + +The switch dials the provided peers in parallel. +The list of peer addresses is randomly shuffled, and for each peer a routine is +spawned. +Each routine sleeps for a random interval, up to 3 seconds, then invokes the +`DialPeerWithAddress` method that actually dials the peer. + +### Reconnect to peer + +The `reconnectToPeer` method is invoked when a connection attempt to a peer fails, +and the peer is configured as a persistent peer. + +The `reconnecting` synchronized map keeps the peer's in this state, identified +by their IDs (string). +This should ensure that a single instance of this method is running at any time. +The peer is kept in this map while this method is running for it: it is set on +the beginning, and removed when the method returns for whatever reason. +If the peer is already in the `reconnecting` state, nothing is done. + +The remaining of the method performs multiple connection attempts to the peer, +via `DialPeerWithAddress` method. +If a connection attempt succeeds, the methods returns and the routine finishes. +The same applies when an `ErrCurrentlyDialingOrExistingAddress` error is +returned by the dialing method, as it indicates that peer is already connected +or that another routine is attempting to (re)connect to it. + +A first set of connection attempts is done at (about) regular intervals. +More precisely, between two attempts, the switch waits for a interval of +`reconnectInterval`, hard-coded to 5 seconds, plus a random jitter up to +`dialRandomizerIntervalMilliseconds`, hard-coded to 3 seconds. +At most `reconnectAttempts`, hard-coded to 20, are made using this +regular-interval approach. + +A second set of connection attempts is done with exponentially increasing +intervals. +The base interval `reconnectBackOffBaseSeconds` is hard-coded to 3 seconds, +which is also the increasing factor. +The exponentially increasing dialing interval is adjusted as well by a random +jitter up to `dialRandomizerIntervalMilliseconds`. +At most `reconnectBackOffAttempts`, hard-coded to 10, are made using this approach. + +> Note: the first sleep interval, to which a random jitter is applied, is 1, +> not `reconnectBackOffBaseSeconds`, as the first exponent is `0`... + +## Accepting peers + +The `acceptRoutine` method is a persistent routine that handles connections +accepted by the transport configured for the switch. + +The [`Accept`](./transport.md#accept) method of the configured transport +returns a `Peer` with which an inbound connection was established. +The switch accepts a new peer if the maximum number of inbound peers was not +reached, or if the peer was configured as an _unconditional peer_. +The maximum number of inbound peers is determined by the `MaxNumInboundPeers` +configuration parameter, whose default value is `40`. + +If accepted, the peer is added to the switch using the [`addPeer`](#add-peer) method. +If the switch does not accept the established incoming connection, or if the +`addPeer` method returns an error, the switch invokes the transport's +[`Cleanup`](./transport.md#cleanup) method to clean any resources associated +with the peer. + +The transport's `Accept` method can also return a number of errors. +Errors of `ErrRejected` or `ErrFilterTimeout` types are ignored, +an `ErrTransportClosed` causes the accepted routine to be interrupted, +while other errors cause the routine to panic. + +> TODO: which errors can cause the routine to panic? + +## Add peer + +The `addPeer` method adds a peer to the switch, +either after dialing (by `addOutboundPeerWithConfig`, called by `DialPeerWithAddress`) +a peer and establishing an outbound connection, +or after accepting (`acceptRoutine`) a peer and establishing an inbound connection. + +The first step is to invoke the `filterPeer` method. +It checks whether the peer is already in the set of connected peers, +and whether any of the configured `peerFilter` methods reject the peer. +If the peer is already present or it is rejected by any filter, the `addPeer` +method fails and returns an error. + +Then, the new peer is started, added to the set of connected peers, and added +to all reactors. +More precisely, first the new peer's information is first provided to every +reactor (`InitPeer` method). +Next, the peer's sending and receiving routines are started, and the peer is +added to set of connected peers. +These two operations can fail, causing `addPeer` to return an error. +Then, in the absence of previous errors, the peer is added to every reactor (`AddPeer` method). + +> Adding the peer to the peer set returns a `ErrSwitchDuplicatePeerID` error +> when a peer with the same ID is already presented. +> +> TODO: Starting a peer could be reduced as starting the MConn with that peer? + +## Stop peer + +There are two methods for stopping a peer, namely disconnecting from it, and +removing it from the table of connected peers. + +The `StopPeerForError` method is invoked to stop a peer due to an external +error, which is provided to method as a generic "reason". + +The `StopPeerGracefully` method stops a peer in the absence of errors or, more +precisely, not providing to the switch any "reason" for that. + +In both cases the `Peer` instance is stopped, the peer is removed from all +registered reactors, and finally from the list of connected peers. + +> Issue is mentioned in +> the internal `stopAndRemovePeer` method explaining why removing the peer from +> the list of connected peers is the last action taken. + +When there is a "reason" for stopping the peer (`StopPeerForError` method) +and the peer is a persistent peer, the method creates a routine to attempt +reconnecting to the peer address, using the `reconnectToPeer` method. +If the peer is an outbound peer, the peer's address is know, since the switch +has dialed the peer. +Otherwise, the peer address is retrieved from the `NodeInfo` instance from the +connection handshake. + +## Add reactor + +The `AddReactor` method registers a `Reactor` to the switch. + +The reactor is associated to the set of channel ids it employs. +Two reactors (in the same node) cannot share the same channel id. + +There is a call back to the reactor, in which the switch passes itself to the +reactor. + +## Remove reactor + +The `RemoveReactor` method unregisters a `Reactor` from the switch. + +The reactor is disassociated from the set of channel ids it employs. + +There is a call back to the reactor, in which the switch passes `nil` to the +reactor. + +## OnStart + +This is a `BaseService` method. + +All registered reactors are started. + +The switch's `acceptRoutine` is started. + +## OnStop + +This is a `BaseService` method. + +All (connected) peers are stopped and removed from the peer's list using the +`stopAndRemovePeer` method. + +All registered reactors are stopped. + +## Broadcast + +This method broadcasts a message on a channel, by sending the message in +parallel to all connected peers. + +The method spawns a thread for each connected peer, invoking the `Send` method +provided by each `Peer` instance with the provided message and channel ID. +The return value (a boolean) of these calls are redirected to a channel that is +returned by the method. + +> TODO: detail where this method is invoked: +> +> - By the consensus protocol, in `broadcastNewRoundStepMessage`, +> `broadcastNewValidBlockMessage`, and `broadcastHasVoteMessage` +> - By the state sync protocol diff --git a/cometbft/v0.38/spec/p2p/implementation/transport.md b/cometbft/v0.38/spec/p2p/implementation/transport.md new file mode 100644 index 00000000..20d4db87 --- /dev/null +++ b/cometbft/v0.38/spec/p2p/implementation/transport.md @@ -0,0 +1,222 @@ +# Transport + +The transport establishes secure and authenticated connections with peers. + +The transport [`Dial`](#dial)s peer addresses to establish outbound connections, +and [`Listen`](#listen)s in a configured network address +to [`Accept`](#accept) inbound connections from peers. + +The transport establishes raw TCP connections with peers +and [upgrade](#connection-upgrade) them into authenticated secret connections. +The established secret connection is then wrapped into `Peer` instance, which +is returned to the caller, typically the [switch](./switch.md). + +## Dial + +The `Dial` method is used by the switch to establish an outbound connection with a peer. +It is a synchronous method, which blocks until a connection is established or an error occurs. +The method returns an outbound `Peer` instance wrapping the established connection. + +The transport first dials the provided peer's address to establish a raw TCP connection. +The dialing maximum duration is determined by `dialTimeout`, hard-coded to 1 second. +The established raw connection is then submitted to a set of [filters](#connection-filtering), +which can reject it. +If the connection is not rejected, it is recorded in the table of established connections. + +The established raw TCP connection is then [upgraded](#connection-upgrade) into +an authenticated secret connection. +This procedure should ensure, in particular, that the public key of the remote peer +matches the ID of the dialed peer, which is part of peer address provided to this method. +In the absence of errors, +the established secret connection (`conn.SecretConnection` type) +and the information about the peer (`NodeInfo` record) retrieved and verified +during the version handshake, +are wrapped into an outbound `Peer` instance and returned to the switch. + +## Listen + +The `Listen` method produces a TCP listener instance for the provided network +address, and spawns an `acceptPeers` routine to handle the raw connections +accepted by the listener. +The `NetAddress` method exports the listen address configured for the transport. + +The maximum number of simultaneous incoming connections accepted by the listener +is bound to `MaxNumInboundPeer` plus the configured number of unconditional peers, +using the `MultiplexTransportMaxIncomingConnections` option, +in the node [initialization](https://github.com/cometbft/cometbft/blob/v0.34.x/node/node.go#L563). + +This method is called when a node is [started](https://github.com/cometbft/cometbft/blob/v0.34.x/node/node.go#L974). +In case of errors, the `acceptPeers` routine is not started and the error is returned. + +## Accept + +The `Accept` method returns to the switch inbound connections established with a peer. +It is a synchronous method, which blocks until a connection is accepted or an error occurs. +The method returns an inbound `Peer` instance wrapping the established connection. + +The transport handles incoming connections in the `acceptPeers` persistent routine. +This routine is started by the [`Listen`](#listen) method +and accepts raw connections from a TCP listener. +A new routine is spawned for each accepted connection. +The raw connection is submitted to a set of [filters](#connection-filtering), +which can reject it. +If the connection is not rejected, it is recorded in the table of established connections. + +The established raw TCP connection is then [upgraded](#connection-upgrade) into +an authenticated secret connection. +The established secret connection (`conn.SecretConnection` type), +the information about the peer (`NodeInfo` record) retrieved and verified +during the version handshake, +as well any error returned in this process are added to a queue of accepted connections. +This queue is consumed by the `Accept` method. + +> Handling accepted connection asynchronously was introduced due to this issue: +> + +## Connection Filtering + +The `filterConn` method is invoked for every new raw connection established by the transport. +Its main goal is avoid the transport to maintain duplicated connections with the same peer. +It also runs a set of configured connection filters. + +The transports keeps a table `conns` of established connections. +The table maps the remote address returned by a generic connection to a list of +IP addresses, to which the connection remote address is resolved. +If the remote address of the new connection is already present in the table, +the connection is rejected. +Otherwise, the connection's remote address is resolved into a list of IPs, +which are recorded in the established connections table. + +The connection and the resolved IPs are then passed through a set of connection filters, +configured via the `MultiplexTransportConnFilters` transport option. +The maximum duration for the filters execution, which is performed in parallel, +is determined by `filterTimeout`. +Its default value is 5 seconds, +which can be changed using the `MultiplexTransportFilterTimeout` transport option. + +If the connection and the resolved remote addresses are not filtered out, +the transport registers them into the `conns` table and returns. + +In case of errors, the connection is removed from the table of established +connections and closed. + +### Errors + +If the address of the new connection is already present in the `conns` table, +an `ErrRejected` error with the `isDuplicate` reason is returned. + +If the IP resolution of the connection's remote address fails, +an `AddrError` or `DNSError` error is returned. + +If any of the filters reject the connection, +an `ErrRejected` error with the `isRejected` reason is returned. + +If the filters execution times out, +an `ErrFilterTimeout` error is returned. + +## Connection Upgrade + +The `upgrade` method is invoked for every new raw connection established by the +transport that was not [filtered out](#connection-filtering). +It upgrades an established raw TCP connection into a secret authenticated +connection, and validates the information provided by the peer. + +This is a complex procedure, that can be summarized by the following three +message exchanges between the node and the new peer: + +1. Encryption: the nodes produce ephemeral key pairs and exchange ephemeral + public keys, from which are derived: (i) a pair of secret keys used to + encrypt the data exchanged between the nodes, and (ii) a challenge message. +1. Authentication: the nodes exchange their persistent public keys and a + signature of the challenge message produced with the their persistent + private keys. This allows validating the peer's persistent public key, + which plays the role of node ID. +1. Version handshake: nodes exchange and validate each other `NodeInfo` records. + This records contain, among other fields, their node IDs, the network/chain + ID they are part of, and the list of supported channel IDs. + +Steps (1) and (2) are implemented in the `conn` package. +In case of success, they produce the secret connection that is actually used by +the node to communicate with the peer. +An overview of this procedure, which implements the station-to-station (STS) +[protocol][sts-paper] ([PDF][sts-paper-pdf]), can be found [here][peer-sts]. +The maximum duration for establishing a secret connection with the peer is +defined by `handshakeTimeout`, hard-coded to 3 seconds. + +The established secret connection stores the persistent public key of the peer, +which has been validated via the challenge authentication of step (2). +If the connection being upgraded is an outbound connection, i.e., if the node has +dialed the peer, the dialed peer's ID is compared to the peer's persistent public key: +if they do not match, the connection is rejected. +This verification is not performed in the case of inbound (accepted) connections, +as the node does not know a priori the remote node's ID. + +Step (3), the version handshake, is performed by the transport. +Its maximum duration is also defined by `handshakeTimeout`, hard-coded to 3 seconds. +The version handshake retrieves the `NodeInfo` record of the new peer, +which can be rejected for multiple reasons, listed [here][peer-handshake]. + +If the connection upgrade succeeds, the method returns the established secret +connection, an instance of `conn.SecretConnection` type, +and the `NodeInfo` record of the peer. + +In case of errors, the connection is removed from the table of established +connections and closed. + +### Errors + +The timeouts for steps (1) and (2), and for step (3), are configured as the +deadline for operations on the TCP connection that is being upgraded. +If this deadline it is reached, the connection produces an +`os.ErrDeadlineExceeded` error, returned by the corresponding step. + +Any error produced when establishing a secret connection with the peer (steps 1 and 2) or +during the version handshake (step 3), including timeouts, +is encapsulated into an `ErrRejected` error with reason `isAuthFailure` and returned. + +If the upgraded connection is an outbound connection, and the peer ID learned in step (2) +does not match the dialed peer's ID, +an `ErrRejected` error with reason `isAuthFailure` is returned. + +If the peer's `NodeInfo` record, retrieved in step (3), is invalid, +or if reports a node ID that does not match peer ID learned in step (2), +an `ErrRejected` error with reason `isAuthFailure` is returned. +If it reports a node ID equals to the local node ID, +an `ErrRejected` error with reason `isSelf` is returned. +If it is not compatible with the local `NodeInfo`, +an `ErrRejected` error with reason `isIncompatible` is returned. + +## Close + +The `Close` method closes the TCP listener created by the `Listen` method, +and sends a signal for interrupting the `acceptPeers` routine. + +This method is called when a node is [stopped](https://github.com/cometbft/cometbft/blob/v0.34.x/node/node.go#L1023). + +## Cleanup + +The `Cleanup` method receives a `Peer` instance, +and removes the connection established with a peer from the table of established connections. +It also invokes the `Peer` interface method to close the connection associated with a peer. + +It is invoked when the connection with a peer is closed. + +## Supported channels + +The `AddChannel` method registers a channel in the transport. + +The channel ID is added to the list of supported channel IDs, +stored in the local `NodeInfo` record. + +The `NodeInfo` record is exchanged with peers in the version handshake. +For this reason, this method is not invoked with a started transport. + +> The only call to this method is performed in the `CustomReactors` constructor +> option of a node, i.e., before the node is started. +> Note that the default list of supported channel IDs, including the default reactors, +> is provided to the transport as its original `NodeInfo` record. + +[peer-sts]: ../legacy-docs/peer.md#authenticated-encryption-handshake +[peer-handshake]: ../legacy-docs/peer.md#cometbft-version-handshake +[sts-paper]: https://link.springer.com/article/10.1007/BF00124891 +[sts-paper-pdf]: https://github.com/tendermint/tendermint/blob/0.1/docs/sts-final.pdf diff --git a/cometbft/v0.38/spec/p2p/implementation/types.md b/cometbft/v0.38/spec/p2p/implementation/types.md new file mode 100644 index 00000000..cef26329 --- /dev/null +++ b/cometbft/v0.38/spec/p2p/implementation/types.md @@ -0,0 +1,233 @@ +# Types adopted in the p2p implementation + +This document lists the packages and source files, excluding test units, that +implement the p2p layer, and summarizes the main types they implement. +Types play the role of classes in Go. + +The reference version for this documentation is the branch +[`v0.34.x`](https://github.com/cometbft/cometbft/tree/v0.34.x/p2p). + +State of August 2022. + +## Package `p2p` + +Implementation of the p2p layer of CometBFT. + +### `base_reactor.go` + +`Reactor` interface. + +`BaseReactor` implements `Reactor`. + +**Not documented yet**. + +### `conn_set.go` + +`ConnSet` interface, a "lookup table for connections and their ips". + +Internal type `connSet` implements the `ConnSet` interface. + +Used by the [transport](#transportgo) to store connected peers. + +### `errors.go` + +Defines several error types. + +`ErrRejected` enumerates a number of reason for which a peer was rejected. +Mainly produced by the [transport](#transportgo), +but also by the [switch](#switchgo). + +`ErrSwitchDuplicatePeerID` is produced by the `PeerSet` used by the [switch](#switchgo). + +`ErrSwitchConnectToSelf` is handled by the [switch](#switchgo), +but currently is not produced outside tests. + +`ErrSwitchAuthenticationFailure` is handled by the [PEX reactor](#pex_reactorgo), +but currently is not produced outside tests. + +`ErrTransportClosed` is produced by the [transport](#transportgo) +and handled by the [switch](#switchgo). + +`ErrNetAddressNoID`, `ErrNetAddressInvalid`, and `ErrNetAddressLookup` +are parsing a string to create an instance of `NetAddress`. +It can be returned in the setup of the [switch](#switchgo) +and of the [PEX reactor](#pex_reactorgo), +as well when the [transport](#transportgo) validates a `NodeInfo`, as part of +the connection handshake. + +`ErrCurrentlyDialingOrExistingAddress` is produced by the [switch](#switchgo), +and handled by the switch and the [PEX reactor](#pex_reactorgo). + +### `fuzz.go` + +For testing purposes. + +`FuzzedConnection` wraps a `net.Conn` and injects random delays. + +### `key.go` + +`NodeKey` is the persistent key of a node, namely its private key. + +The `ID` of a node is a string representing the node's public key. + +### `metrics.go` + +Prometheus `Metrics` exposed by the p2p layer. + +### `netaddress.go` + +Type `NetAddress` contains the `ID` and the network address (IP and port) of a node. + +The API of the [address book](#addrbookgo) receives and returns `NetAddress` instances. + +This source file was adapted from [`btcd`](https://github.com/btcsuite/btcd), +a Go implementation of Bitcoin. + +### `node_info.go` + +Interface `NodeInfo` stores the basic information about a node exchanged with a +peer during the handshake. + +It is implemented by `DefaultNodeInfo` type. + +The [switch](#switchgo) stores the local `NodeInfo`. + +The `NodeInfo` of connected peers is produced by the +[transport](#transportgo) during the handshake, and stored in [`Peer`](#peergo) instances. + +### `peer.go` + +Interface `Peer` represents a connected peer. + +It is implemented by the internal `peer` type. + +The [transport](#transportgo) API methods return `Peer` instances, +wrapping established secure connection with peers. + +The [switch](#switchgo) API methods receive `Peer` instances. +The switch stores connected peers in a `PeerSet`. + +The [`Reactor`](#base_reactorgo) methods, invoked by the switch, receive `Peer` instances. + +### `peer_set.go` + +Interface `IPeerSet` offers methods to access a table of [`Peer`](#peergo) instances. + +Type `PeerSet` implements a thread-safe table of [`Peer`](#peergo) instances, +used by the [switch](#switchgo). + +The switch provides limited access to this table by returing a `IPeerSet` +instance, used by the [PEX reactor](#pex_reactorgo). + +### `switch.go` + +Documented in [switch](./switch.md). + +The `Switch` implements the [peer manager](./peer_manager.md) role for inbound peers. + +[`Reactor`](#base_reactorgo)s have access to the `Switch` and may invoke its methods. +This includes the [PEX reactor](#pex_reactorgo). + +### `transport.go` + +Documented in [transport](./transport.md). + +The `Transport` interface is implemented by `MultiplexTransport`. + +The [switch](#switchgo) contains a `Transport` and uses it to establish +connections with peers. + +### `types.go` + +Aliases for p2p's `conn` package types. + +## Package `p2p.conn` + +Implements the connection between CometBFT nodes, +which is encrypted, authenticated, and multiplexed. + +### `connection.go` + +Implements the `MConnection` type and the `Channel` abstraction. + +A `MConnection` multiplexes a generic network connection (`net.Conn`) into +multiple independent `Channel`s, used by different [`Reactor`](#base_reactorgo)s. + +A [`Peer`](#peergo) stores the `MConnection` instance used to interact with a +peer, which multiplex a [`SecretConnection`](#secret_connectiongo). + +### `conn_go110.go` + +Support for go 1.10. + +### `secret_connection.go` + +Implements the `SecretConnection` type, which is an encrypted authenticated +connection built atop a raw network (TCP) connection. + +A [`Peer`](#peergo) stores the `SecretConnection` established by the transport, +which is the underlying connection multiplexed by [`MConnection`](#connectiongo). + +As briefly documented in the [transport](./transport.md#Connection-Upgrade), +a `SecretConnection` implements the Station-To-Station (STS) protocol. + +The `SecretConnection` type implements the `net.Conn` interface, +which is a generic network connection. + +## Package `p2p.mock` + +Mock implementations of [`Peer`](#peergo) and [`Reactor`](#base_reactorgo) interfaces. + +## Package `p2p.mocks` + +Code generated by `mockery`. + +## Package `p2p.pex` + +Implementation of the [PEX reactor](./pex.md). + +### `addrbook.go` + +Documented in [address book](./addressbook.md). + +This source file was adapted from [`btcd`](https://github.com/btcsuite/btcd), +a Go implementation of Bitcoin. + +### `errors.go` + +A number of errors produced and handled by the [address book](#addrbookgo). + +`ErrAddrBookNilAddr` is produced by the address book, but handled (logged) by +the [PEX reactor](#pex_reactorgo). + +`ErrUnsolicitedList` is produced and handled by the [PEX protocol](#pex_reactorgo). + +### `file.go` + +Implements the [address book](#addrbookgo) persistence. + +### `known_address.go` + +Type `knownAddress` represents an address stored in the [address book](#addrbookgo). + +### `params.go` + +Constants used by the [address book](#addrbookgo). + +### `pex_reactor.go` + +Implementation of the [PEX reactor](./pex.md), which is a [`Reactor`](#base_reactorgo). + +This includes the implementation of the [PEX protocol](./pex-protocol.md) +and of the [peer manager](./peer_manager.md) role for outbound peers. + +The PEX reactor also manages an [address book](#addrbookgo) instance. + +## Package `p2p.trust` + +Go documentation of `Metric` type: + +> // Metric - keeps track of peer reliability +> // See cometbft/docs/architecture/adr-006-trust-metric.md for details + +Not imported by any other CometBFT source file. diff --git a/cometbft/v0.38/spec/p2p/legacy-docs/Overview.mdx b/cometbft/v0.38/spec/p2p/legacy-docs/Overview.mdx new file mode 100644 index 00000000..e2895b5e --- /dev/null +++ b/cometbft/v0.38/spec/p2p/legacy-docs/Overview.mdx @@ -0,0 +1,16 @@ +--- +order: 1 +title: Legacy Docs +--- + +# Legacy Docs + +This section contains useful information. However, part of this content is redundant, being more comprehensively covered +in more recent documents, and some implementation details might be outdated +(see issue [#981](https://github.com/cometbft/cometbft/issues/981)). + +- [Messages](/cometbft/v0.38/spec/p2p/legacy-docs/messages/Overview) +- [P2P Config](/cometbft/v0.38/spec/p2p/legacy-docs/P2P-Config) +- [P2P Multiplex Connection](/cometbft/v0.38/spec/p2p/legacy-docs/P2P-Multiplex-Connection) +- [Peer Discovery](/cometbft/v0.38/spec/p2p/legacy-docs/Peer-Discovery) +- [Peers](/cometbft/v0.38/spec/p2p/legacy-docs/Peers) diff --git a/cometbft/v0.38/spec/p2p/legacy-docs/P2P-Config.mdx b/cometbft/v0.38/spec/p2p/legacy-docs/P2P-Config.mdx new file mode 100644 index 00000000..34383e62 --- /dev/null +++ b/cometbft/v0.38/spec/p2p/legacy-docs/P2P-Config.mdx @@ -0,0 +1,53 @@ +--- +order: 1 +--- + +# P2P Config + +Here we describe configuration options around the Peer Exchange. +These can be set using flags or via the `$CMTHOME/config/config.toml` file. + +## Seed Mode + +`--p2p.seed_mode` + +The node operates in seed mode. In seed mode, a node continuously crawls the network for peers, +and upon incoming connection shares some peers and disconnects. + +## Seeds + +`--p2p.seeds “id100000000000000000000000000000000@1.2.3.4:26656,id200000000000000000000000000000000@2.3.4.5:4444”` + +Dials these seeds when we need more peers. They should return a list of peers and then disconnect. +If we already have enough peers in the address book, we may never need to dial them. + +## Persistent Peers + +`--p2p.persistent_peers “id100000000000000000000000000000000@1.2.3.4:26656,id200000000000000000000000000000000@2.3.4.5:26656”` + +Dial these peers and auto-redial them if the connection fails. +These are intended to be trusted persistent peers that can help +anchor us in the p2p network. The auto-redial uses exponential +backoff and will give up after a day of trying to connect. + +But If `persistent_peers_max_dial_period` is set greater than zero, +pause between each dial to each persistent peer will not exceed `persistent_peers_max_dial_period` +during exponential backoff and we keep trying again without giving up + +**Note:** If `seeds` and `persistent_peers` intersect, +the user will be warned that seeds may auto-close connections +and that the node may not be able to keep the connection persistent. + +## Private Peers + +`--p2p.private_peer_ids “id100000000000000000000000000000000,id200000000000000000000000000000000”` + +These are IDs of the peers that we do not add to the address book or gossip to +other peers. They stay private to us. + +## Unconditional Peers + +`--p2p.unconditional_peer_ids “id100000000000000000000000000000000,id200000000000000000000000000000000”` + +These are IDs of the peers which are allowed to be connected by both inbound or outbound regardless of +`max_num_inbound_peers` or `max_num_outbound_peers` of user's node reached or not. diff --git a/cometbft/v0.38/spec/p2p/legacy-docs/P2P-Multiplex-Connection.mdx b/cometbft/v0.38/spec/p2p/legacy-docs/P2P-Multiplex-Connection.mdx new file mode 100644 index 00000000..eb255a44 --- /dev/null +++ b/cometbft/v0.38/spec/p2p/legacy-docs/P2P-Multiplex-Connection.mdx @@ -0,0 +1,115 @@ +--- +order: 1 +--- + +# P2P Multiplex Connection + +## MConnection + +`MConnection` is a multiplex connection that supports multiple independent streams +with distinct quality of service guarantees atop a single TCP connection. +Each stream is known as a `Channel` and each `Channel` has a globally unique _byte id_. +Each `Channel` also has a relative priority that determines the quality of service +of the `Channel` compared to other `Channel`s. +The _byte id_ and the relative priorities of each `Channel` are configured upon +initialization of the connection. + +The `MConnection` supports three packet types: + +- Ping +- Pong +- Msg + +### Ping and Pong + +The ping and pong messages consist of writing a single byte to the connection; 0x1 and 0x2, respectively. + +When we haven't received any messages on an `MConnection` in time `pingTimeout`, we send a ping message. +When a ping is received on the `MConnection`, a pong is sent in response only if there are no other messages +to send and the peer has not sent us too many pings (TODO). + +If a pong or message is not received in sufficient time after a ping, the peer is disconnected from. + +### Msg + +Messages in channels are chopped into smaller `msgPacket`s for multiplexing. + +```go +type msgPacket struct { + ChannelID byte + EOF byte // 1 means message ends here. + Bytes []byte +} +``` + +The `msgPacket` is serialized using [Proto3](https://developers.google.com/protocol-buffers/docs/proto3). +The received `Bytes` of a sequential set of packets are appended together +until a packet with `EOF=1` is received, then the complete serialized message +is returned for processing by the `onReceive` function of the corresponding channel. + +### Multiplexing + +Messages are sent from a single `sendRoutine`, which loops over a select statement and results in the sending +of a ping, a pong, or a batch of data messages. The batch of data messages may include messages from multiple channels. +Message bytes are queued for sending in their respective channel, with each channel holding one unsent message at a time. +Messages are chosen for a batch one at a time from the channel with the lowest ratio of recently sent bytes to channel priority. + +## Sending Messages + +There are two methods for sending messages: + +```go +func (m MConnection) Send(chID byte, msg interface{}) bool {} +func (m MConnection) TrySend(chID byte, msg interface{}) bool {} +``` + +`Send(chID, msg)` is a blocking call that waits until `msg` is successfully queued +for the channel with the given id byte `chID`. The message `msg` is serialized +using protobuf marshalling. + +`TrySend(chID, msg)` is a nonblocking call that queues the message msg in the channel +with the given id byte chID if the queue is not full; otherwise it returns false immediately. + +`Send()` and `TrySend()` are also exposed for each `Peer`. + +## Peer + +Each peer has one `MConnection` instance, and includes other information such as whether the connection +was outbound, whether the connection should be recreated if it closes, various identity information about the node, +and other higher level thread-safe data used by the reactors. + +## Switch/Reactor + +The `Switch` handles peer connections and exposes an API to receive incoming messages +on `Reactors`. Each `Reactor` is responsible for handling incoming messages of one +or more `Channels`. So while sending outgoing messages is typically performed on the peer, +incoming messages are received on the reactor. + +```go +// Declare a MyReactor reactor that handles messages on MyChannelID. +type MyReactor struct{} + +func (reactor MyReactor) GetChannels() []*ChannelDescriptor { + return []*ChannelDescriptor{ChannelDescriptor{ID:MyChannelID, Priority: 1}} +} + +func (reactor MyReactor) Receive(chID byte, peer *Peer, msgBytes []byte) { + r, n, err := bytes.NewBuffer(msgBytes), new(int64), new(error) + msgString := ReadString(r, n, err) + fmt.Println(msgString) +} + +// Other Reactor methods omitted for brevity +... + +switch := NewSwitch([]Reactor{MyReactor{}}) + +... + +// Send a random message to all outbound connections +for _, peer := range switch.Peers().List() { + if peer.IsOutbound() { + peer.Send(MyChannelID, "Here's a random message") + } +} +``` diff --git a/cometbft/v0.38/spec/p2p/legacy-docs/Peer-Discovery.mdx b/cometbft/v0.38/spec/p2p/legacy-docs/Peer-Discovery.mdx new file mode 100644 index 00000000..492fb56b --- /dev/null +++ b/cometbft/v0.38/spec/p2p/legacy-docs/Peer-Discovery.mdx @@ -0,0 +1,69 @@ +--- +order: 1 +--- + +# Peer Discovery + +A CometBFT P2P network has different kinds of nodes with different requirements for connectivity to one another. +This document describes what kind of nodes CometBFT should enable and how they should work. + +## Seeds + +Seeds are the first point of contact for a new node. +They return a list of known active peers and then disconnect. + +Seeds should operate full nodes with the PEX reactor in a "crawler" mode +that continuously explores to validate the availability of peers. + +Seeds should only respond with some top percentile of the best peers it knows about. + +## New Full Node + +A new node needs a few things to connect to the network: + +- a list of seeds, which can be provided to CometBFT via config file or flags, + or hardcoded into the software by in-process apps +- a `ChainID`, also called `Network` at the p2p layer +- a recent block height, H, and hash, HASH for the blockchain. + +The values `H` and `HASH` must be received and corroborated by means external to CometBFT, and specific to the user - ie. via the user's trusted social consensus. +This requirement to validate `H` and `HASH` out-of-band and via social consensus +is the essential difference in security models between Proof-of-Work and Proof-of-Stake blockchains. + +With the above, the node then queries some seeds for peers for its chain, +dials those peers, and runs the CometBFT protocols with those it successfully connects to. + +When the peer catches up to height H, it ensures the block hash matches HASH. +If not, CometBFT will exit, and the user must try again - either they are connected +to bad peers or their social consensus is invalid. + +## Restarted Full Node + +A node checks its address book on startup and attempts to connect to peers from there. +If it can't connect to any peers after some time, it falls back to the seeds to find more. + +Restarted full nodes can run the `blockchain` or `consensus` reactor protocols to sync up +to the latest state of the blockchain from wherever they were last. +In a Proof-of-Stake context, if they are sufficiently far behind (greater than the length +of the unbonding period), they will need to validate a recent `H` and `HASH` out-of-band again +so they know they have synced the correct chain. + +## Validator Node + +A validator node is a node that interfaces with a validator signing key. +These nodes require the highest security, and should not accept incoming connections. +They should maintain outgoing connections to a controlled set of "Sentry Nodes" that serve +as their proxy shield to the rest of the network. + +Validators that know and trust each other can accept incoming connections from one another and maintain direct private connectivity via VPN. + +## Sentry Node + +Sentry nodes are guardians of a validator node and provide it access to the rest of the network. +They should be well connected to other full nodes on the network. +Sentry nodes may be dynamic, but should maintain persistent connections to some evolving random subset of each other. +They should always expect to have direct incoming connections from the validator node and its backup(s). +They do not report the validator node's address in the PEX and +they may be more strict about the quality of peers they keep. + +Sentry nodes belonging to validators that trust each other may wish to maintain persistent connections via VPN with one another, but only report each other sparingly in the PEX. diff --git a/cometbft/v0.38/spec/p2p/legacy-docs/Peers.mdx b/cometbft/v0.38/spec/p2p/legacy-docs/Peers.mdx new file mode 100644 index 00000000..69217a62 --- /dev/null +++ b/cometbft/v0.38/spec/p2p/legacy-docs/Peers.mdx @@ -0,0 +1,132 @@ +--- +order: 1 +--- + +# Peers + +This document explains how CometBFT Peers are identified and how they connect to one another. + +## Peer Identity + +CometBFT peers are expected to maintain long-term persistent identities in the form of a public key. +Each peer has an ID defined as `peer.ID == peer.PubKey.Address()`, where `Address` uses the scheme defined in `crypto` package. + +A single peer ID can have multiple IP addresses associated with it, but a node +will only ever connect to one at a time. + +When attempting to connect to a peer, we use the PeerURL: `@:`. +We will attempt to connect to the peer at IP:PORT, and verify, +via authenticated encryption, that it is in possession of the private key +corresponding to ``. This prevents man-in-the-middle attacks on the peer layer. + +## Connections + +All p2p connections use TCP. +Upon establishing a successful TCP connection with a peer, +two handshakes are performed: one for authenticated encryption, and one for CometBFT versioning. +Both handshakes have configurable timeouts (they should complete quickly). + +### Authenticated Encryption Handshake + +CometBFT implements the Station-to-Station protocol +using X25519 keys for Diffie-Helman key-exchange and chacha20poly1305 for encryption. + +Previous versions of this protocol (0.32 and below) suffered from malleability attacks whereas an active man +in the middle attacker could compromise confidentiality as described in [Prime, Order Please! +Revisiting Small Subgroup and Invalid Curve Attacks on +Protocols using Diffie-Hellman](https://eprint.iacr.org/2019/526.pdf). + +We have added dependency on the Merlin a keccak based transcript hashing protocol to ensure non-malleability. + +It goes as follows: + +- generate an ephemeral X25519 keypair +- send the ephemeral public key to the peer +- wait to receive the peer's ephemeral public key +- create a new Merlin Transcript with the string "TENDERMINT_SECRET_CONNECTION_TRANSCRIPT_HASH" +- Sort the ephemeral keys and add the high labeled "EPHEMERAL_UPPER_PUBLIC_KEY" and the low keys labeled "EPHEMERAL_LOWER_PUBLIC_KEY" to the Merlin transcript. +- compute the Diffie-Hellman shared secret using the peers ephemeral public key and our ephemeral private key +- add the DH secret to the transcript labeled DH_SECRET. +- generate two keys to use for encryption (sending and receiving) and a challenge for authentication as follows: + - create a hkdf-sha256 instance with the key being the diffie hellman shared secret, and info parameter as + `TENDERMINT_SECRET_CONNECTION_KEY_AND_CHALLENGE_GEN` + - get 64 bytes of output from hkdf-sha256 + - if we had the smaller ephemeral pubkey, use the first 32 bytes for the key for receiving, the second 32 bytes for sending; else the opposite. +- use a separate nonce for receiving and sending. Both nonces start at 0, and should support the full 96 bit nonce range +- all communications from now on are encrypted in 1400 byte frames (plus encoding overhead), + using the respective secret and nonce. Each nonce is incremented by one after each use. +- we now have an encrypted channel, but still need to authenticate +- extract a 32 bytes challenge from merlin transcript with the label "SECRET_CONNECTION_MAC" +- sign the common challenge obtained from the hkdf with our persistent private key +- send the amino encoded persistent pubkey and signature to the peer +- wait to receive the persistent public key and signature from the peer +- verify the signature on the challenge using the peer's persistent public key + +If this is an outgoing connection (we dialed the peer) and we used a peer ID, +then finally verify that the peer's persistent public key corresponds to the peer ID we dialed, +ie. `peer.PubKey.Address() == `. + +The connection has now been authenticated. All traffic is encrypted. + +Note: only the dialer can authenticate the identity of the peer, +but this is what we care about since when we join the network we wish to +ensure we have reached the intended peer (and are not being MITMd). + +### Peer Filter + +Before continuing, we check if the new peer has the same ID as ourselves or +an existing peer. If so, we disconnect. + +We also check the peer's address and public key against +an optional whitelist which can be managed through the ABCI app - +if the whitelist is enabled and the peer does not qualify, the connection is +terminated. + +### CometBFT Version Handshake + +The CometBFT Version Handshake allows the peers to exchange their NodeInfo: + +```golang +type NodeInfo struct { + Version p2p.Version + ID p2p.ID + ListenAddr string + + Network string + SoftwareVersion string + Channels []int8 + + Moniker string + Other NodeInfoOther +} + +type Version struct { + P2P uint64 + Block uint64 + App uint64 +} + +type NodeInfoOther struct { + TxIndex string + RPCAddress string +} +``` + +The connection is disconnected if: + +- `peer.NodeInfo.ID` is not equal `peerConn.ID` +- `peer.NodeInfo.Version.Block` does not match ours +- `peer.NodeInfo.Network` is not the same as ours +- `peer.Channels` does not intersect with our known Channels. +- `peer.NodeInfo.ListenAddr` is malformed or is a DNS host that cannot be + resolved + +At this point, if we have not disconnected, the peer is valid. +It is added to the switch and hence all reactors via the `AddPeer` method. +Note that each reactor may handle multiple channels. + +## Connection Activity + +Once a peer is added, incoming messages for a given reactor are handled through +that reactor's `Receive` method, and output messages are sent directly by the Reactors +on each peer. A typical reactor maintains per-peer go-routine(s) that handle this. diff --git a/cometbft/v0.38/spec/p2p/legacy-docs/messages/Overview.mdx b/cometbft/v0.38/spec/p2p/legacy-docs/messages/Overview.mdx new file mode 100644 index 00000000..243e0bb6 --- /dev/null +++ b/cometbft/v0.38/spec/p2p/legacy-docs/messages/Overview.mdx @@ -0,0 +1,19 @@ +--- +order: 1 +parent: + title: Messages + order: 1 +--- + +# Messages + +An implementation of the spec consists of many components. While many parts of these components are implementation specific, the p2p messages are not. In this section we will be covering all the p2p messages of components. + +There are two parts to the P2P messages, the message and the channel. The channel is message specific and messages are specific to components of CometBFT. When a node connect to a peer it will tell the other node which channels are available. This notifies the peer what services the connecting node offers. You can read more on channels in [connection.md](/cometbft/v0.38/spec/p2p/legacy-docs/P2P-Multiplex-Connection) + +- [Block Sync](/cometbft/v0.38/spec/p2p/legacy-docs/messages/block-sync) +- [Mempool](/cometbft/v0.38/spec/p2p/legacy-docs/messages/mempool) +- [Evidence](/cometbft/v0.38/spec/p2p/legacy-docs/messages/evidence) +- [State Sync](/cometbft/v0.38/spec/p2p/legacy-docs/messages/state-sync) +- [Pex](/cometbft/v0.38/spec/p2p/legacy-docs/messages/Peer-Exchange) +- [Consensus](/cometbft/v0.38/spec/p2p/legacy-docs/messages/consensus) diff --git a/cometbft/v0.38/spec/p2p/legacy-docs/messages/Peer-Exchange.mdx b/cometbft/v0.38/spec/p2p/legacy-docs/messages/Peer-Exchange.mdx new file mode 100644 index 00000000..cab26ac2 --- /dev/null +++ b/cometbft/v0.38/spec/p2p/legacy-docs/messages/Peer-Exchange.mdx @@ -0,0 +1,76 @@ +--- +order: 6 +--- + +# Peer Exchange + +## Channels + +Pex has one channel. The channel identifier is listed below. + +| Name | Number | +|------------|--------| +| PexChannel | 0 | + +## Message Types + +The current PEX service has two versions. The first uses IP/port pair but since the p2p stack is moving towards a transport agnostic approach, +node endpoints require a `Protocol` and `Path` hence the V2 version uses a [url](https://golang.org/pkg/net/url/#URL) instead. + +### PexRequest + +PexRequest is an empty message requesting a list of peers. + +> EmptyRequest + +### PexResponse + +PexResponse is an list of net addresses provided to a peer to dial. + +| Name | Type | Description | Field Number | +|-------|------------------------------------|------------------------------------------|--------------| +| addresses | repeated [PexAddress](#pexaddress) | List of peer addresses available to dial | 1 | + +### PexAddress + +PexAddress provides needed information for a node to dial a peer. + +| Name | Type | Description | Field Number | +|------|--------|------------------|--------------| +| id | string | NodeID of a peer | 1 | +| ip | string | The IP of a node | 2 | +| port | port | Port of a peer | 3 | + + +### PexRequestV2 + +PexRequest is an empty message requesting a list of peers. + +> EmptyRequest + +### PexResponseV2 + +PexResponse is an list of net addresses provided to a peer to dial. + +| Name | Type | Description | Field Number | +|-------|------------------------------------|------------------------------------------|--------------| +| addresses | repeated [PexAddressV2](#pexresponsev2) | List of peer addresses available to dial | 1 | + +### PexAddressV2 + +PexAddress provides needed information for a node to dial a peer. + +| Name | Type | Description | Field Number | +|------|--------|------------------|--------------| +| url | string | See [golang url](https://golang.org/pkg/net/url/#URL) | 1 | + +### Message + +Message is a [`oneof` protobuf type](https://developers.google.com/protocol-buffers/docs/proto#oneof). The one of consists of two messages. + +| Name | Type | Description | Field Number | +|--------------|---------------------------|------------------------------------------------------|--------------| +| pex_request | [PexRequest](#pexrequest) | Empty request asking for a list of addresses to dial | 1 | +| pex_response | [PexResponse](#pexresponse)| List of addresses to dial | 2 | +| pex_request_v2| [PexRequestV2](#pexrequestv2)| Empty request asking for a list of addresses to dial| 3 | +| pex_response_v2| [PexRespinseV2](#pexresponsev2)| List of addresses to dial | 4 | diff --git a/cometbft/v0.38/spec/p2p/legacy-docs/messages/block-sync.mdx b/cometbft/v0.38/spec/p2p/legacy-docs/messages/block-sync.mdx new file mode 100644 index 00000000..49afcc41 --- /dev/null +++ b/cometbft/v0.38/spec/p2p/legacy-docs/messages/block-sync.mdx @@ -0,0 +1,70 @@ +--- +order: 2 +--- + +# Block Sync + +## Channel + +Block sync has one channel. + +| Name | Number | +|-------------------|--------| +| BlocksyncChannel | 64 | + +## Message Types + +There are multiple message types for Block Sync + +### BlockRequest + +BlockRequest asks a peer for a block at the height specified. + +| Name | Type | Description | Field Number | +|--------|-------|---------------------------|--------------| +| Height | int64 | Height of requested block | 1 | + +### NoBlockResponse + +NoBlockResponse notifies the peer requesting a block that the node does not contain it. + +| Name | Type | Description | Field Number | +|--------|-------|---------------------------|--------------| +| Height | int64 | Height of requested block | 1 | + +### BlockResponse + +BlockResponse contains the block requested. +It also contains an extended commit _iff_ vote extensions are enabled at the block's height. + +| Name | Type | Description | Field Number | +|-----------|----------------------------------------------------------------|---------------------------------|--------------| +| Block | [Block](../../../core/data_structures.md#block) | Requested Block | 1 | +| ExtCommit | [ExtendedCommit](../../../core/data_structures.md#extendedcommit) | Sender's LastCommit information | 2 | + +### StatusRequest + +StatusRequest is an empty message that notifies the peer to respond with the highest and lowest blocks it has stored. + +> Empty message. + +### StatusResponse + +StatusResponse responds to a peer with the highest and lowest heights of any block it has in its blockstore. + +| Name | Type | Description | Field Number | +|--------|-------|-------------------------------------------------------------------|--------------| +| Height | int64 | Current Height of a node | 1 | +| Base | int64 | First known block, if pruning is enabled it will be higher than 1 | 2 | + +### Message + +Message is a [`oneof` protobuf type](https://developers.google.com/protocol-buffers/docs/proto#oneof). The `oneof` consists of five messages. + +| Name | Type | Description | Field Number | +|-------------------|-------------------------------------|--------------------------------------------------------------|--------------| +| block_request | [BlockRequest](#blockrequest) | Request a block from a peer | 1 | +| no_block_response | [NoBlockResponse](#noblockresponse) | Response saying it doe snot have the requested block | 2 | +| block_response | [BlockResponse](#blockresponse) | Response with requested block + (optionally) vote extensions | 3 | +| status_request | [StatusRequest](#statusrequest) | Request the highest and lowest block numbers from a peer | 4 | +| status_response | [StatusResponse](#statusresponse) | Response with the highest and lowest block numbers the store | 5 | diff --git a/cometbft/v0.38/spec/p2p/legacy-docs/messages/consensus.mdx b/cometbft/v0.38/spec/p2p/legacy-docs/messages/consensus.mdx new file mode 100644 index 00000000..f65c68c6 --- /dev/null +++ b/cometbft/v0.38/spec/p2p/legacy-docs/messages/consensus.mdx @@ -0,0 +1,146 @@ +--- +order: 7 +--- + +# Consensus + +## Channel + +Consensus has four separate channels. The channel identifiers are listed below. + +| Name | Number | +|--------------------|--------| +| StateChannel | 32 | +| DataChannel | 33 | +| VoteChannel | 34 | +| VoteSetBitsChannel | 35 | + +## Message Types + +### Proposal + +Proposal is sent when a new block is proposed. It is a suggestion of what the +next block in the blockchain should be. + +| Name | Type | Description | Field Number | +|----------|----------------------------------------------------|----------------------------------------|--------------| +| proposal | [Proposal](../../../core/data_structures.md#proposal) | Proposed Block to come to consensus on | 1 | + +### Vote + +Vote is sent to vote for some block (or to inform others that a process does not vote in the +current round). Vote contains validator's information (validator address and index), height and +round for which the vote is sent, vote type, blockID if process vote for some block (`nil` otherwise) +and a timestamp when the vote is sent. The message is signed by the validator private key. + +| Name | Type | Description | Field Number | +|------|--------------------------------------------|---------------------------|--------------| +| vote | [Vote](../../../core/data_structures.md#vote) | Vote for a proposed Block | 1 | + +### BlockPart + +BlockPart is sent when gossiping a piece of the proposed block. It contains height, round +and the block part. + +| Name | Type | Description | Field Number | +|--------|--------------------------------------------|----------------------------------------|--------------| +| height | int64 | Height of corresponding block. | 1 | +| round | int32 | Round of voting to finalize the block. | 2 | +| part | [Part](../../../core/data_structures.md#part) | A part of the block. | 3 | + +### NewRoundStep + +NewRoundStep is sent for every step transition during the core consensus algorithm execution. +It is used in the gossip part of the CometBFT consensus protocol to inform peers about a current +height/round/step a process is in. + +| Name | Type | Description | Field Number | +|--------------------------|--------|----------------------------------------|--------------| +| height | int64 | Height of corresponding block | 1 | +| round | int32 | Round of voting to finalize the block. | 2 | +| step | uint32 | | 3 | +| seconds_since_start_time | int64 | | 4 | +| last_commit_round | int32 | | 5 | + +### NewValidBlock + +NewValidBlock is sent when a validator observes a valid block B in some round r, +i.e., there is a Proposal for block B and 2/3+ prevotes for the block B in the round r. +It contains height and round in which valid block is observed, block parts header that describes +the valid block and is used to obtain all +block parts, and a bit array of the block parts a process currently has, so its peers can know what +parts it is missing so they can send them. +In case the block is also committed, then IsCommit flag is set to true. + +| Name | Type | Description | Field Number | +|-----------------------|--------------------------------------------------------------|----------------------------------------|--------------| +| height | int64 | Height of corresponding block | 1 | +| round | int32 | Round of voting to finalize the block. | 2 | +| block_part_set_header | [PartSetHeader](../../../core/data_structures.md#partsetheader) | | 3 | +| block_parts | int32 | | 4 | +| is_commit | bool | | 5 | + +### ProposalPOL + +ProposalPOL is sent when a previous block is re-proposed. +It is used to inform peers in what round the process learned for this block (ProposalPOLRound), +and what prevotes for the re-proposed block the process has. + +| Name | Type | Description | Field Number | +|--------------------|----------|-------------------------------|--------------| +| height | int64 | Height of corresponding block | 1 | +| proposal_pol_round | int32 | | 2 | +| proposal_pol | bitarray | | 3 | + +### ReceivedVote + +ReceivedVote is sent to indicate that a particular vote has been received. It contains height, +round, vote type and the index of the validator that is the originator of the corresponding vote. + +| Name | Type | Description | Field Number | +|--------|------------------------------------------------------------------|----------------------------------------|--------------| +| height | int64 | Height of corresponding block | 1 | +| round | int32 | Round of voting to finalize the block. | 2 | +| type | [SignedMessageType](../../../core/data_structures.md#signedmsgtype) | | 3 | +| index | int32 | | 4 | + +### VoteSetMaj23 + +VoteSetMaj23 is sent to indicate that a process has seen +2/3 votes for some BlockID. +It contains height, round, vote type and the BlockID. + +| Name | Type | Description | Field Number | +|--------|------------------------------------------------------------------|----------------------------------------|--------------| +| height | int64 | Height of corresponding block | 1 | +| round | int32 | Round of voting to finalize the block. | 2 | +| type | [SignedMessageType](../../../core/data_structures.md#signedmsgtype) | | 3 | + +### VoteSetBits + +VoteSetBits is sent to communicate the bit-array of votes a process has seen for a given +BlockID. It contains height, round, vote type, BlockID and a bit array of +the votes a process has. + +| Name | Type | Description | Field Number | +|----------|------------------------------------------------------------------|----------------------------------------|--------------| +| height | int64 | Height of corresponding block | 1 | +| round | int32 | Round of voting to finalize the block. | 2 | +| type | [SignedMessageType](../../../core/data_structures.md#signedmsgtype) | | 3 | +| block_id | [BlockID](../../../core/data_structures.md#blockid) | | 4 | +| votes | BitArray | Round of voting to finalize the block. | 5 | + +### Message + +Message is a [`oneof` protobuf type](https://developers.google.com/protocol-buffers/docs/proto#oneof). + +| Name | Type | Description | Field Number | +|-----------------|---------------------------------|----------------------------------------|--------------| +| new_round_step | [NewRoundStep](#newroundstep) | Height of corresponding block | 1 | +| new_valid_block | [NewValidBlock](#newvalidblock) | Round of voting to finalize the block. | 2 | +| proposal | [Proposal](#proposal) | | 3 | +| proposal_pol | [ProposalPOL](#proposalpol) | | 4 | +| block_part | [BlockPart](#blockpart) | | 5 | +| vote | [Vote](#vote) | | 6 | +| received_vote | [ReceivedVote](#receivedvote) | | 7 | +| vote_set_maj23 | [VoteSetMaj23](#votesetmaj23) | | 8 | +| vote_set_bits | [VoteSetBits](#votesetbits) | | 9 | diff --git a/cometbft/v0.38/spec/p2p/legacy-docs/messages/evidence.mdx b/cometbft/v0.38/spec/p2p/legacy-docs/messages/evidence.mdx new file mode 100644 index 00000000..7db104b3 --- /dev/null +++ b/cometbft/v0.38/spec/p2p/legacy-docs/messages/evidence.mdx @@ -0,0 +1,23 @@ +--- +order: 3 +--- + +# Evidence + +## Channel + +Evidence has one channel. The channel identifier is listed below. + +| Name | Number | +|-----------------|--------| +| EvidenceChannel | 56 | + +## Message Types + +### EvidenceList + +EvidenceList consists of a list of verified evidence. This evidence will already have been propagated throughout the network. EvidenceList is used in two places, as a p2p message and within the block [block](../../../core/data_structures.md#block) as well. + +| Name | Type | Description | Field Number | +|----------|-------------------------------------------------------------|------------------------|--------------| +| evidence | repeated [Evidence](../../../core/data_structures.md#evidence) | List of valid evidence | 1 | diff --git a/cometbft/v0.38/spec/p2p/legacy-docs/messages/mempool.mdx b/cometbft/v0.38/spec/p2p/legacy-docs/messages/mempool.mdx new file mode 100644 index 00000000..8f3925ca --- /dev/null +++ b/cometbft/v0.38/spec/p2p/legacy-docs/messages/mempool.mdx @@ -0,0 +1,33 @@ +--- +order: 4 +--- +# Mempool + +## Channel + +Mempool has one channel. The channel identifier is listed below. + +| Name | Number | +|----------------|--------| +| MempoolChannel | 48 | + +## Message Types + +There is currently only one message that Mempool broadcasts and receives over +the p2p gossip network (via the reactor): `TxsMessage` + +### Txs + +A list of transactions. These transactions have been checked against the application for validity. This does not mean that the transactions are valid, it is up to the application to check this. + +| Name | Type | Description | Field Number | +|------|----------------|----------------------|--------------| +| txs | repeated bytes | List of transactions | 1 | + +### Message + +Message is a [`oneof` protobuf type](https://developers.google.com/protocol-buffers/docs/proto#oneof). The one of consists of one message [`Txs`](#txs). + +| Name | Type | Description | Field Number | +|------|-------------|-----------------------|--------------| +| txs | [Txs](#txs) | List of transactions | 1 | diff --git a/cometbft/v0.38/spec/p2p/legacy-docs/messages/state-sync.mdx b/cometbft/v0.38/spec/p2p/legacy-docs/messages/state-sync.mdx new file mode 100644 index 00000000..30657ecb --- /dev/null +++ b/cometbft/v0.38/spec/p2p/legacy-docs/messages/state-sync.mdx @@ -0,0 +1,132 @@ +--- +order: 5 +--- + +# State Sync + +## Channels + +State sync has four distinct channels. The channel identifiers are listed below. + +| Name | Number | +|-------------------|--------| +| SnapshotChannel | 96 | +| ChunkChannel | 97 | +| LightBlockChannel | 98 | +| ParamsChannel | 99 | + +## Message Types + +### SnapshotRequest + +When a new node begin state syncing, it will ask all peers it encounters if it has any +available snapshots: + +| Name | Type | Description | Field Number | +|----------|--------|-------------|--------------| + +### SnapShotResponse + +The receiver will query the local ABCI application via `ListSnapshots`, and send a message +containing snapshot metadata (limited to 4 MB) for each of the 10 most recent snapshots: and stored at the application layer. When a peer is starting it will request snapshots. + +| Name | Type | Description | Field Number | +|----------|--------|-----------------------------------------------------------|--------------| +| height | uint64 | Height at which the snapshot was taken | 1 | +| format | uint32 | Format of the snapshot. | 2 | +| chunks | uint32 | How many chunks make up the snapshot | 3 | +| hash | bytes | Arbitrary snapshot hash | 4 | +| metadata | bytes | Arbitrary application data. **May be non-deterministic.** | 5 | + +### ChunkRequest + +The node running state sync will offer these snapshots to the local ABCI application via +`OfferSnapshot` ABCI calls, and keep track of which peers contain which snapshots. Once a snapshot +is accepted, the state syncer will request snapshot chunks from appropriate peers: + +| Name | Type | Description | Field Number | +|--------|--------|-------------------------------------------------------------|--------------| +| height | uint64 | Height at which the chunk was created | 1 | +| format | uint32 | Format chosen for the chunk. **May be non-deterministic.** | 2 | +| index | uint32 | Index of the chunk within the snapshot. | 3 | + +### ChunkResponse + +The receiver will load the requested chunk from its local application via `LoadSnapshotChunk`, +and respond with it (limited to 16 MB): + +| Name | Type | Description | Field Number | +|---------|--------|-------------------------------------------------------------|--------------| +| height | uint64 | Height at which the chunk was created | 1 | +| format | uint32 | Format chosen for the chunk. **May be non-deterministic.** | 2 | +| index | uint32 | Index of the chunk within the snapshot. | 3 | +| hash | bytes | Arbitrary snapshot hash | 4 | +| missing | bool | Arbitrary application data. **May be non-deterministic.** | 5 | + +Here, `Missing` is used to signify that the chunk was not found on the peer, since an empty +chunk is a valid (although unlikely) response. + +The returned chunk is given to the ABCI application via `ApplySnapshotChunk` until the snapshot +is restored. If a chunk response is not returned within some time, it will be re-requested, +possibly from a different peer. + +The ABCI application is able to request peer bans and chunk refetching as part of the ABCI protocol. + +### LightBlockRequest + +To verify state and to provide state relevant information for consensus, the node will ask peers for +light blocks at specified heights. + +| Name | Type | Description | Field Number | +|----------|--------|----------------------------|--------------| +| height | uint64 | Height of the light block | 1 | + +### LightBlockResponse + +The receiver will retrieve and construct the light block from both the block and state stores. The +receiver will verify the data by comparing the hashes and store the header, commit and validator set +if necessary. The light block at the height of the snapshot will be used to verify the `AppHash`. + +| Name | Type | Description | Field Number | +|---------------|---------------------------------------------------------|--------------------------------------|--------------| +| light_block | [LightBlock](../../../core/data_structures.md#lightblock) | Light block at the height requested | 1 | + +State sync will use [light client verification](../../../light-client/verification/README.md) to verify +the light blocks. + +If no state sync is in progress (i.e. during normal operation), any unsolicited response messages +are discarded. + +### ParamsRequest + +In order to build the state, the state provider will request the params at the height of the snapshot and use the header to verify it. + +| Name | Type | Description | Field Number | +|----------|--------|----------------------------|--------------| +| height | uint64 | Height of the consensus params | 1 | + + +### ParamsResponse + +A reciever to the request will use the state store to fetch the consensus params at that height and return it to the sender. + +| Name | Type | Description | Field Number | +|----------|--------|---------------------------------|--------------| +| height | uint64 | Height of the consensus params | 1 | +| consensus_params | [ConsensusParams](../../../core/data_structures.md#consensusparams) | Consensus params at the height requested | 2 | + + +### Message + +Message is a [`oneof` protobuf type](https://developers.google.com/protocol-buffers/docs/proto#oneof). The `oneof` consists of eight messages. + +| Name | Type | Description | Field Number | +|----------------------|--------------------------------------------|----------------------------------------------|--------------| +| snapshots_request | [SnapshotRequest](#snapshotrequest) | Request a recent snapshot from a peer | 1 | +| snapshots_response | [SnapshotResponse](#snapshotresponse) | Respond with the most recent snapshot stored | 2 | +| chunk_request | [ChunkRequest](#chunkrequest) | Request chunks of the snapshot. | 3 | +| chunk_response | [ChunkRequest](#chunkresponse) | Response of chunks used to recreate state. | 4 | +| light_block_request | [LightBlockRequest](#lightblockrequest) | Request a light block. | 5 | +| light_block_response | [LightBlockResponse](#lightblockresponse) | Respond with a light block | 6 | +| params_request | [ParamsRequest](#paramsrequest) | Request the consensus params at a height. | 7 | +| params_response | [ParamsResponse](#paramsresponse) | Respond with the consensus params | 8 | diff --git a/cometbft/v0.38/spec/p2p/reactor-api/API-for-Reactors.mdx b/cometbft/v0.38/spec/p2p/reactor-api/API-for-Reactors.mdx new file mode 100644 index 00000000..fd22b87a --- /dev/null +++ b/cometbft/v0.38/spec/p2p/reactor-api/API-for-Reactors.mdx @@ -0,0 +1,335 @@ +--- +order: 3 +--- + +# API for Reactors + +This document describes the API provided by the p2p layer to the protocol +layer, namely to the registered reactors. + +This API consists of two interfaces: the one provided by the `Switch` instance, +and the ones provided by multiple `Peer` instances, one per connected peer. +The `Switch` instance is provided to every reactor as part of the reactor's +[registration procedure][reactor-registration]. +The multiple `Peer` instances are provided to every registered reactor whenever +a [new connection with a peer][reactor-addpeer] is established. + +> **Note** +> +> The practical reasons that lead to the interface to be provided in two parts, +> `Switch` and `Peer` instances are discussed in more datail in the +> [knowledge-base repository](https://github.com/cometbft/knowledge-base/blob/main/p2p/reactors/switch-peer.md). + +## `Switch` API + +The [`Switch`][switch-type] is the central component of the p2p layer +implementation. It manages all the reactors running in a node and keeps track +of the connections with peers. +The table below summarizes the interaction of the standard reactors with the `Switch`: + +| `Switch` API method | consensus | block sync | state sync | mempool | evidence | PEX | +|--------------------------------------------|-----------|------------|------------|---------|-----------|-------| +| `Peers() IPeerSet` | x | x | | | | x | +| `NumPeers() (int, int, int)` | | x | | | | x | +| `Broadcast(Envelope) chan bool` | x | x | x | | | | +| `MarkPeerAsGood(Peer)` | x | | | | | | +| `StopPeerForError(Peer, interface{})` | x | x | x | x | x | x | +| `StopPeerGracefully(Peer)` | | | | | | x | +| `Reactor(string) Reactor` | | x | | | | | + +The above list is not exhaustive as it does not include all the `Switch` methods +invoked by the PEX reactor, a special component that should be considered part +of the p2p layer. This document does not cover the operation of the PEX reactor +as a connection manager. + +### Peers State + +The first two methods in the switch API allow reactors to query the state of +the p2p layer: the set of connected peers. +~~~ + + func (sw *Switch) Peers() IPeerSet +~~~ + +The `Peers()` method returns the current set of connected peers. +The returned `IPeerSet` is an immutable concurrency-safe copy of this set. +Observe that the `Peer` handlers returned by this method were previously +[added to the reactor][reactor-addpeer] via the `InitPeer(Peer)` method, +but not yet removed via the `RemovePeer(Peer)` method. +Thus, a priori, reactors should already have this information. +~~~ + + func (sw *Switch) NumPeers() (outbound, inbound, dialing int) +~~~ + +The `NumPeers()` method returns the current number of connected peers, +distinguished between `outbound` and `inbound` peers. +An `outbound` peer is a peer the node has dialed to, while an `inbound` peer is +a peer the node has accepted a connection from. +The third field `dialing` reports the number of peers to which the node is +currently attempting to connect, so not (yet) connected peers. + +> **Note** +> +> The third field returned by `NumPeers()`, the number of peers in `dialing` +> state, is not an information that should regard the protocol layer. +> In fact, with the exception of the PEX reactor, which can be considered part +> of the p2p layer implementation, no standard reactor actually uses this +> information, that could be removed when this interface is refactored. + +### Broadcast + +The switch provides, mostly for historical or retro-compatibility reasons, +a method for sending a message to all connected peers: +~~~ + + func (sw *Switch) Broadcast(e Envelope) chan bool +~~~ + +The `Broadcast()` method is not blocking and returns a channel of booleans. +For every connected `Peer`, it starts a background thread for sending the +message to that peer, using the `Peer.Send()` method +(which is blocking, as detailed in [Send Methods](#send-methods)). +The result of each unicast send operation (success or failure) is added to the +returned channel, which is closed when all operations are completed. + +> **Note** +> +> - The current _implementation_ of the `Switch.Broadcast(Envelope)` method is +> not efficient, as the marshalling of the provided message is performed as +> part of the `Peer.Send(Envelope)` helper method, that is, once per +> connected peer. +> - The return value of the broadcast method is not considered by any of the +> standard reactors that employ the method. One of the reasons is that is is +> not possible to associate each of the boolean outputs added to the +> returned channel to a peer. + +### Vetting Peers + +The p2p layer relies on the registered reactors to gauge the _quality_ of peers. +The following method can be invoked by a reactor to inform the p2p layer that a +peer has presented a "good" behaviour. +This information is registered in the node's address book and influences the +operation of the Peer Exchange (PEX) protocol, as node discovery adopts a bias +towards "good" peers: +~~~ + + func (sw *Switch) MarkPeerAsGood(peer Peer) +~~~ + +At the moment, it is up to the consensus reactor to vet a peer. +In the current logic, a peer is marked as good whenever the consensus protocol +collects a multiple of `votesToContributeToBecomeGoodPeer = 10000` useful votes +or `blocksToContributeToBecomeGoodPeer = 10000` useful block parts from that peer. +By "useful", the consensus implementation considers messages that are valid and +that are received by the node when the node is expected for such information, +which excludes duplicated or late received messages. + +> **Note** +> +> The switch doesn't currently provide a method to mark a peer as a bad peer. +> In fact, the peer quality management is really implemented in the current +> version of the p2p layer. +> This topic is being discussed in the [knowledge-base repository](https://github.com/cometbft/knowledge-base/blob/main/p2p/reactors/peer-quality.md). + +### Stopping Peers + +Reactors can instruct the p2p layer to disconnect from a peer. +Using the p2p layer's nomenclature, the reactor requests a peer to be stopped. +The peer's send and receive routines are in fact stopped, interrupting the +communication with the peer. +The `Peer` is then [removed from every registered reactor][reactor-removepeer], +using the `RemovePeer(Peer)` method, and from the set of connected peers. +~~~ + + func (sw *Switch) StopPeerForError(peer Peer, reason interface{}) +~~~ + +All the standard reactors employ the above method for disconnecting from a peer +in case of errors. +These are errors that occur when processing a message received from a `Peer`. +The produced `error` is provided to the method as the `reason`. + +The `StopPeerForError()` method has an important *caveat*: if the peer to be +stopped is configured as a _persistent peer_, the switch will attempt +reconnecting to that same peer. +While this behaviour makes sense when the method is invoked by other components +of the p2p layer (e.g., in the case of communication errors), it does not make +sense when it is invoked by a reactor. + +> **Note** +> +> A more comprehensive discussion regarding this topic can be found on the +> [knowledge-base repository](https://github.com/cometbft/knowledge-base/blob/main/p2p/reactors/stop-peer.md). + + func (sw *Switch) StopPeerGracefully(peer Peer) + +The second method instructs the switch to disconnect from a peer for no +particular reason. +This method is only adopted by the PEX reactor of a node operating in _seed mode_, +as seed nodes disconnect from a peer after exchanging peer addresses with it. + +### Reactors Table + +The switch keeps track of all registered reactors, indexed by unique reactor names. +A reactor can therefore use the switch to access another `Reactor` from its `name`: +~~~ + + func (sw *Switch) Reactor(name string) Reactor +~~~ + +This method is currently only used by the Block Sync reactor to access the +Consensus reactor implementation, from which it uses the exported +`SwitchToConsensus()` method. +While available, this inter-reactor interaction approach is discouraged and +should be avoided, as it violates the assumption that reactors are independent. + + +## `Peer` API + +The [`Peer`][peer-interface] interface represents a connected peer. +A `Peer` instance encapsulates a multiplex connection that implements the +actual communication (sending and receiving messages) with a peer. +When a connection is established with a peer, the `Switch` provides the +corresponding `Peer` instance to all registered reactors. +From this point, reactors can use the methods of the new `Peer` instance. + +The table below summarizes the interaction of the standard reactors with +connected peers, with the `Peer` methods used by them: + +| `Peer` API method | consensus | block sync | state sync | mempool | evidence | PEX | +|--------------------------------------------|-----------|------------|------------|---------|-----------|-------| +| `ID() ID` | x | x | x | x | x | x | +| `IsRunning() bool` | x | | | x | x | | +| `Quit() <-chan struct{}` | | | | x | x | | +| `Get(string) interface{}` | x | | | x | x | | +| `Set(string, interface{})` | x | | | | | | +| `Send(Envelope) bool` | x | x | x | x | x | x | +| `TrySend(Envelope) bool` | x | x | | | | | + +The above list is not exhaustive as it does not include all the `Peer` methods +invoked by the PEX reactor, a special component that should be considered part +of the p2p layer. This document does not cover the operation of the PEX reactor +as a connection manager. + +### Identification + +Nodes in the p2p network are configured with a unique cryptographic key pair. +The public part of this key pair is verified when establishing a connection +with the peer, as part of the authentication handshake, and constitutes the +peer's `ID`: +~~~ + + func (p Peer) ID() p2p.ID +~~~ + +Observe that each time the node connects to a peer (e.g., after disconnecting +from it), a new (distinct) `Peer` handler is provided to the reactors via +`InitPeer(Peer)` method. +In fact, the `Peer` handler is associated to a _connection_ with a peer, not to +the actual _node_ in the network. +To keep track of actual peers, the unique peer `p2p.ID` provided by the above +method should be employed. + +### Peer state + +The switch starts the peer's send and receive routines before adding the peer +to every registered reactor using the `AddPeer(Peer)` method. +The reactors then usually start routines to interact with the new connected +peer using the received `Peer` handler. +For these routines it is useful to check whether the peer is still connected +and its send and receive routines are still running: +~~~ + + func (p Peer) IsRunning() bool + func (p Peer) Quit() <-chan struct{} +~~~ + +The above two methods provide the same information about the state of a `Peer` +instance in two different ways. +Both of them are defined in the [`Service`][service-interface] interface. +The `IsRunning()` method is synchronous and returns whether the peer has been +started and has not been stopped. +The `Quit()` method returns a channel that is closed when the peer is stopped; +it is an asynchronous state query. + +### Key-value store + +Each `Peer` instance provides a synchronized key-value store that allows +sharing peer-specific state between reactors: + +~~~ + + func (p Peer) Get(key string) interface{} + func (p Peer) Set(key string, data interface{}) +~~~ + +This key-value store can be seen as an asynchronous mechanism to exchange the +state of a peer between reactors. +In the current use-case of this mechanism, the Consensus reactor populates the +key-value store with a `PeerState` instance for each connected peer. +The Consensus reactor routines interacting with a peer read and update the +shared peer state. +The Evidence and Mempool reactors, in their turn, periodically query the +key-value store of each peer for retrieving, in particular, the last height +reported by the peer. +This information, produced by the Consensus reactor, influences the interaction +of these two reactors with their peers. + +> **Note** +> +> More details of how this key-value store is used to share state between reactors can be found on the +> [knowledge-base repository](https://github.com/cometbft/knowledge-base/blob/main/p2p/reactors/peer-kvstore.md). + +### Send methods + +Finally, a `Peer` instance allows a reactor to send messages to companion +reactors running at that peer. +This is ultimately the goal of the switch when it provides `Peer` instances to +the registered reactors. +There are two methods for sending messages: +~~~ + + func (p Peer) Send(e Envelope) bool + func (p Peer) TrySend(e Envelope) bool +~~~ + +The two message-sending methods receive an `Envelope`, whose content should be +set as follows: + +- `ChannelID`: the channel the message should be sent through, which defines + the reactor that will process the message; +- `Src`: this field represents the source of an incoming message, which is + irrelevant for outgoing messages; +- `Message`: the actual message's payload, which is marshalled using protocol buffers. + +The two message-sending methods attempt to add the message (`e.Payload`) to the +send queue of the peer's destination channel (`e.ChannelID`). +There is a send queue for each registered channel supported by the peer, and +each send queue has a capacity. +The capacity of the send queues for each channel are [configured][reactor-channels] +by reactors via the corresponding `ChannelDescriptor`. + +The two message-sending methods return whether it was possible to enqueue +the marshalled message to the channel's send queue. +The most common reason for these methods to return `false` is the channel's +send queue being full. +Further reasons for returning `false` are: the peer being stopped, providing a +non-registered channel ID, or errors when marshalling the message's payload. + +The difference between the two message-sending methods is _when_ they return `false`. +The `Send()` method is a _blocking_ method, it returns `false` if the message +could not be enqueued, because the channel's send queue is still full, after a +10-second _timeout_. +The `TrySend()` method is a _non-blocking_ method, it _immediately_ returns +`false` when the channel's send queue is full. + +[peer-interface]: https://github.com/cometbft/cometbft/blob/v0.38.x/p2p/peer.go +[service-interface]: https://github.com/cometbft/cometbft/blob/v0.38.x/libs/service/service.go +[switch-type]: https://github.com/cometbft/cometbft/blob/v0.38.x/p2p/switch.go + +[reactor-interface]: https://github.com/cometbft/cometbft/blob/v0.38.x/p2p/base_reactor.go +[reactor-registration]: ./reactor.md#registration +[reactor-channels]: ./reactor.md#registration +[reactor-addpeer]: ./reactor.md#peer-management +[reactor-removepeer]: ./reactor.md#stop-peer diff --git a/cometbft/v0.38/spec/p2p/reactor-api/Reactor-Api.mdx b/cometbft/v0.38/spec/p2p/reactor-api/Reactor-Api.mdx new file mode 100644 index 00000000..a7862fae --- /dev/null +++ b/cometbft/v0.38/spec/p2p/reactor-api/Reactor-Api.mdx @@ -0,0 +1,234 @@ +--- +order: 2 +--- + +# Reactor API + +A component has to implement the [`p2p.Reactor` interface][reactor-interface] +in order to use communication services provided by the p2p layer. +This interface is currently the main source of documentation for a reactor. + +The goal of this document is to specify the behaviour of the p2p communication +layer when interacting with a reactor. +So while the [`Reactor interface`][reactor-interface] declares the methods +invoked and determines what the p2p layer expects from a reactor, +this documentation focuses on the **temporal behaviour** that a reactor implementation +should expect from the p2p layer. (That is, in which orders the functions may be called) + +This specification is accompanied by the [`reactor.qnt`](./reactor.qnt) file, +a more comprehensive model of the reactor's operation written in +[Quint][quint-repo], an executable specification language. +The methods declared in the [`Reactor`][reactor-interface] interface are +modeled in Quint, in the form of `pure def` methods, providing some examples of +how they should be implemented. +The behaviour of the p2p layer when interacting with a reactor, by invoking the +interface methods, is modeled in the form of state transitions, or `action`s in +the Quint nomenclature. + +## Overview + +The following _grammar_ is a simplified representation of the expected sequence of calls +from the p2p layer to a reactor. +Note that the grammar represents events referring to a _single reactor_, while +the p2p layer supports the execution of multiple reactors. +For a more detailed representation of the sequence of calls from the p2p layer +to reactors, please refer to the companion Quint model. + +While useful to provide an overview of the operation of a reactor, +grammars have some limitations in terms of the behaviour they can express. +For instance, the following grammar only represents the management of _a single peer_, +namely of a peer with a given ID which can connect, disconnect, and reconnect +multiple times to the node. +The p2p layer and every reactor should be able to handle multiple distinct peers in parallel. +This means that multiple occurrences of non-terminal `peer-management` of the +grammar below can "run" independently and in parallel, each one referring and +producing events associated to a different peer: + +```abnf +start = registration on-start *peer-management on-stop +registration = get-channels set-switch + +; Refers to a single peer, a reactor must support multiple concurrent peers +peer-management = init-peer start-peer stop-peer +start-peer = [*receive] (connected-peer / start-error) +connected-peer = add-peer *receive +stop-peer = [peer-error] remove-peer + +; Service interface +on-start = %s"OnStart()" +on-stop = %s"OnStop()" +; Reactor interface +get-channels = %s"GetChannels()" +set-switch = %s"SetSwitch(*Switch)" +init-peer = %s"InitPeer(Peer)" +add-peer = %s"AddPeer(Peer)" +remove-peer = %s"RemovePeer(Peer, reason)" +receive = %s"Receive(Envelope)" + +; Errors, for reference +start-error = %s"log(Error starting peer)" +peer-error = %s"log(Stopping peer for error)" +``` + +The grammar is written in case-sensitive Augmented Backus–Naur form (ABNF, +specified in [IETF RFC 7405](https://datatracker.ietf.org/doc/html/rfc7405)). +It is inspired on the grammar produced to specify the interaction of CometBFT +with an ABCI++ application, available [here](../../abci/abci%2B%2B_comet_expected_behavior.md). + +## Registration + +To become a reactor, a component has first to implement the +[`Reactor`][reactor-interface] interface, +then to register the implementation with the p2p layer, using the +`Switch.AddReactor(name string, reactor Reactor)` method, +with a global unique `name` for the reactor. + +The registration must happen before the node, in general, and the p2p layer, +in particular, are started. +In other words, there is no support for registering a reactor on a running node: +reactors must be registered as part of the setup of a node. + +```abnf +registration = get-channels set-switch +``` + +The p2p layer retrieves from the reactor a list of channels the reactor is +responsible for, using the `GetChannels()` method. +The reactor implementation should thereafter expect the delivery of every +message received by the p2p layer in the informed channels. + +The second method `SetSwitch(Switch)` concludes the handshake between the +reactor and the p2p layer. +The `Switch` is the main component of the p2p layer, being responsible for +establishing connections with peers and routing messages. +The `Switch` instance provides a number of methods for all registered reactors, +documented in the companion [API for Reactors](./p2p-api.md#switch-api) document. + +## Service interface + +A reactor must implement the [`Service`](https://github.com/cometbft/cometbft/blob/v0.38.x/libs/service/service.go) interface, +in particular, a startup `OnStart()` and a shutdown `OnStop()` methods: + +```abnf +start = registration on-start *peer-management on-stop +``` + +As part of the startup of a node, all registered reactors are started by the p2p layer. +And when the node is shut down, all registered reactors are stopped by the p2p layer. +Observe that the `Service` interface specification establishes that a service +can be started and stopped only once. +So before being started or once stopped by the p2p layer, the reactor should +not expect any interaction. + +## Peer management + +The core of a reactor's operation is the interaction with peers or, more +precisely, with companion reactors operating on the same channels in peers connected to the node. +The grammar extract below represents the interaction of the reactor with a +single peer: + +```abnf +; Refers to a single peer, a reactor must support multiple concurrent peers +peer-management = init-peer start-peer stop-peer +``` + +The p2p layer informs all registered reactors when it establishes a connection +with a `Peer`, using the `InitPeer(Peer)` method. +When this method is invoked, the `Peer` has not yet been started, namely the +routines for sending messages to and receiving messages from the peer are not running. +This method should be used to initialize state or data related to the new +peer, but not to interact with it. + +The next step is to start the communication routines with the new `Peer`. +As detailed in the following, this procedure may or may not succeed. +In any case, the peer is eventually stopped, which concludes the management of +that `Peer` instance. + +## Start peer + +Once `InitPeer(Peer)` is invoked for every registered reactor, the p2p layer starts the peer's +communication routines and adds the `Peer` to the set of connected peers. +If both steps are concluded without errors, the reactor's `AddPeer(Peer)` is invoked: + +```abnf +start-peer = [*receive] (connected-peer / start-error) +connected-peer = add-peer *receive +``` + +In case of errors, a message is logged informing that the p2p layer failed to start the peer. +This is not a common scenario and it is only expected to happen when +interacting with a misbehaving or slow peer. A practical example is reported on this +[issue](https://github.com/tendermint/tendermint/pull/9500). + +It is up to the reactor to define how to process the `AddPeer(Peer)` event. +The typical behavior is to start routines that, given some conditions or events, +send messages to the added peer, using the provided `Peer` instance. +The companion [API for Reactors](./p2p-api.md#peer-api) documents the methods +provided by `Peer` instances, available from when they are added to the reactors. + +## Stop Peer + +The p2p layer informs all registered reactors when it disconnects from a `Peer`, +using the `RemovePeer(Peer, reason)` method: + +```abnf +stop-peer = [peer-error] remove-peer +``` + +This method is invoked after the p2p layer has stopped peer's send and receive routines. +Depending of the `reason` for which the peer was stopped, different log +messages can be produced. +After removing a peer from all reactors, the `Peer` instance is also removed from +the set of connected peers. +This enables the same peer to reconnect and `InitPeer(Peer)` to be invoked for +the new connection. + +From the removal of a `Peer` , the reactor should not receive any further message +from the peer and must not try sending messages to the removed peer. +This usually means stopping the routines that were started by the companion +`Add(Peer)` method. + +## Receive messages + +The main duty of a reactor is to handle incoming messages on the channels it +has registered with the p2p layer. + +The _pre-condition_ for receiving a message from a `Peer` is that the p2p layer +has previously invoked `InitPeer(Peer)`. +This means that the reactor must be able to receive a message from a `Peer` +_before_ `AddPeer(Peer)` is invoked. +This happens because the peer's send and receive routines are started before, +and should be already running when the p2p layer adds the peer to every +registered reactor. + +```abnf +start-peer = [*receive] (connected-peer / start-error) +connected-peer = add-peer *receive +``` + +The most common scenario, however, is to start receiving messages from a peer +after `AddPeer(Peer)` is invoked. +An arbitrary number of messages can be received, until the peer is stopped and +`RemovePeer(Peer)` is invoked. + +When a message is received from a connected peer on any of the channels +registered by the reactor, the p2p layer will deliver the message to the +reactor via the `Receive(Envelope)` method. +The message is packed into an `Envelope` that contains: + +- `ChannelID`: the channel the message belongs to +- `Src`: the source `Peer` handler, from which the message was received +- `Message`: the actual message's payload, unmarshalled using protocol buffers + +Two important observations regarding the implementation of the `Receive` method: + +1. Concurrency: the implementation should consider concurrent invocations of + the `Receive` method carrying messages from different peers, as the + interaction with different peers is independent and messages can be received in parallel. +1. Non-blocking: the implementation of the `Receive` method is expected not to block, + as it is invoked directly by the receive routines. + In other words, while `Receive` does not return, other messages from the + same sender are not delivered to any reactor. + +[reactor-interface]: https://github.com/cometbft/cometbft/blob/v0.38.x/p2p/base_reactor.go +[quint-repo]: https://github.com/informalsystems/quint diff --git a/cometbft/v0.38/spec/p2p/reactor-api/Reactors.mdx b/cometbft/v0.38/spec/p2p/reactor-api/Reactors.mdx new file mode 100644 index 00000000..b8c27a58 --- /dev/null +++ b/cometbft/v0.38/spec/p2p/reactor-api/Reactors.mdx @@ -0,0 +1,47 @@ +--- +order: 1 +--- + +# Reactors + +Reactor is the generic name for a component that employs the p2p communication layer. + +This section documents the interaction of the p2p communication layer with the +reactors. +The diagram below summarizes this interaction, namely the **northbound interface** +of the p2p communication layer, representing some relevant event flows: + +![P2P Reactors](../images/p2p-reactors.png) + +Each of the protocols running a CometBFT node implements a reactor and registers +the implementation with the p2p layer. +The p2p layer provides network events to the registered reactors, the main +two being new connections with peers and received messages. +The reactors provide to the p2p layer messages to be sent to +peers and commands to control the operation of the p2p layer. + +It is worth noting that the components depicted in the diagram below run +multiple routines and that the illustrated actions happen in parallel. +For instance, the connection establishment routines run in parallel, invoking +the depicted `AddPeer` method concurrently. +Once a connection is fully established, each `Peer` instance runs a send and a +receive routines. +The send routine collects messages from multiple reactors to a peer, packaging +then into raw messages which are transmitted to the peer. +The receive routine processes incoming messages and forwards them to the +destination reactors, invoking the depicted `Receive` methods. +In addition, the reactors run multiple routines for interacting +with the peers (for example, to send messages to them) or with the `Switch`. + +The remaining of the documentation is organized as follows: + +- [Reactor API](./reactor.md): documents the [`p2p.Reactor`][reactor-interface] + interface and specifies the behaviour of the p2p layer when interacting with + a reactor. + In other words, the interaction of the p2p layer with the protocol layer (bottom-up). + +- [P2P API](./p2p-api.md): documents the interface provided by the p2p + layer to the reactors, through the `Switch` and `Peer` abstractions. + In other words, the interaction of the protocol layer with the p2p layer (top-down). + +[reactor-interface]: https://github.com/cometbft/cometbft/blob/v0.38.x/p2p/base_reactor.go diff --git a/cometbft/v0.38/spec/p2p/reactor-api/reactor.qnt b/cometbft/v0.38/spec/p2p/reactor-api/reactor.qnt new file mode 100644 index 00000000..002c5702 --- /dev/null +++ b/cometbft/v0.38/spec/p2p/reactor-api/reactor.qnt @@ -0,0 +1,276 @@ +// -*- mode: Bluespec; -*- +/* + * Reactor is responsible for handling incoming messages on one or more + * Channel. Switch calls GetChannels when reactor is added to it. When a new + * peer joins our node, InitPeer and AddPeer are called. RemovePeer is called + * when the peer is stopped. Receive is called when a message is received on a + * channel associated with this reactor. + */ +// Code: https://github.com/cometbft/cometbft/blob/main/p2p/base_reactor.go +module reactor { + + // Unique ID of a node. + type NodeID = str + + /* + * Peer is an interface representing a peer connected on a reactor. + */ + type Peer = { + ID: NodeID, + + // Other fields can be added to represent the p2p operation. + } + + // Byte ID used by channels, must be globally unique. + type Byte = str + + // Channel configuration. + type ChannelDescriptor = { + ID: Byte, + Priority: int, + } + + /* + * Envelope contains a message with sender routing info. + */ + type Envelope = { + Src: Peer, // Sender + Message: str, // Payload + ChannelID: Byte, + } + + // A Routine is used to interact with an active Peer. + type Routine = { + name: str, + peer: Peer, + } + + type ReactorState = { + // Peers that have been initialized but not yet removed. + // The reactor should expect receiving messages from them. + peers: Set[NodeID], + + // The reactor runs multiple routines. + routines: Set[Routine], + + // Values: init -> registered -> running -> stopped + state: str, + + // Name with which the reactor was registered. + name: str, + + // Channels the reactor is responsible for. + channels: Set[ChannelDescriptor], + } + + // Produces a new, uninitialized reactor. + pure def NewReactor(): ReactorState = { + { + peers: Set(), + routines: Set(), + state: "init", + name: "", + channels: Set(), + } + } + + // Pure definitions below represent the `p2p.Reactor` interface methods: + + /* + * GetChannels returns the list of MConnection.ChannelDescriptor. Make sure + * that each ID is unique across all the reactors added to the switch. + */ + pure def GetChannels(s: ReactorState): Set[ChannelDescriptor] = { + s.channels // Static list, configured at initialization. + } + + /* + * SetSwitch allows setting a switch. + */ + pure def SetSwitch(s: ReactorState, switch: bool): ReactorState = { + s.with("state", "registered") + } + + /* + * Start the service. + * If it's already started or stopped, will return an error. + */ + pure def OnStart(s: ReactorState): ReactorState = { + // Startup procedures should come here. + s.with("state", "running") + } + + /* + * Stop the service. + * If it's already stopped, will return an error. + */ + pure def OnStop(s: ReactorState): ReactorState = { + // Shutdown procedures should come here. + s.with("state", "stopped") + } + + /* + * InitPeer is called by the switch before the peer is started. Use it to + * initialize data for the peer (e.g. peer state). + */ + pure def InitPeer(s: ReactorState, peer: Peer): (ReactorState, Peer) = { + // This method can update the received peer, which is returned. + val updatedPeer = peer + (s.with("peers", s.peers.union(Set(peer.ID))), updatedPeer) + } + + /* + * AddPeer is called by the switch after the peer is added and successfully + * started. Use it to start goroutines communicating with the peer. + */ + pure def AddPeer(s: ReactorState, peer: Peer): ReactorState = { + // This method can be used to start routines to handle the peer. + // Below an example of an arbitrary 'ioRoutine' routine. + val startedRoutines = Set( {name: "ioRoutine", peer: peer} ) + s.with("routines", s.routines.union(startedRoutines)) + } + + /* + * RemovePeer is called by the switch when the peer is stopped (due to error + * or other reason). + */ + pure def RemovePeer(s: ReactorState, peer: Peer, reason: str): ReactorState = { + // This method should stop routines created by `AddPeer(Peer)`. + val stoppedRoutines = s.routines.filter(r => r.peer.ID == peer.ID) + s.with("peers", s.peers.exclude(Set(peer.ID))) + .with("routines", s.routines.exclude(stoppedRoutines)) + } + + /* + * Receive is called by the switch when an envelope is received from any connected + * peer on any of the channels registered by the reactor. + */ + pure def Receive(s: ReactorState, e: Envelope): ReactorState = { + // This method should process the message payload: e.Message. + s + } + + // Global state + + // Reactors are uniquely identified by their names. + var reactors: str -> ReactorState + + // Reactor (name) assigned to each channel ID. + var reactorsByCh: Byte -> str + + // Helper action to (only) update the state of a given reactor. + action updateReactorTo(reactor: ReactorState): bool = all { + reactors' = reactors.set(reactor.name, reactor), + reactorsByCh' = reactorsByCh + } + + // State transitions performed by the p2p layer, invoking `p2p.Reactor` methods: + + // Code: Switch.AddReactor(name string, reactor Reactor) + action register(name: str, reactor: ReactorState): bool = all { + reactor.state == "init", + // Assign the reactor as responsible for its channel IDs, which + // should not be already assigned to another reactor. + val chIDs = reactor.GetChannels().map(c => c.ID) + all { + size(chIDs.intersect(reactorsByCh.keys())) == 0, + reactorsByCh' = reactorsByCh.keys().union(chIDs). + mapBy(id => if (id.in(chIDs)) name + else reactorsByCh.get(id)), + }, + // Register the reactor by its name, which must be unique. + not(name.in(reactors.keys())), + reactors' = reactors.put(name, + reactor.SetSwitch(true).with("name", name)) + } + + // Code: Switch.OnStart() + action start(reactor: ReactorState): bool = all { + reactor.state == "registered", + updateReactorTo(reactor.OnStart()) + } + + // Code: Switch.addPeer(p Peer): preamble + action initPeer(reactor: ReactorState, peer: Peer): bool = all { + reactor.state == "running", + not(peer.ID.in(reactor.peers)), + updateReactorTo(reactor.InitPeer(peer)._1) + } + + // Code: Switch.addPeer(p Peer): conclusion + action addPeer(reactor: ReactorState, peer: Peer): bool = all { + reactor.state == "running", + peer.ID.in(reactor.peers), // InitPeer(peer) and not RemovePeer(peer) + reactor.routines.filter(r => r.peer.ID == peer.ID).size() == 0, + updateReactorTo(reactor.AddPeer(peer)) + } + + // Code: Switch.stopAndRemovePeer(peer Peer, reason interface{}) + action removePeer(reactor: ReactorState, peer: Peer, reason: str): bool = all { + reactor.state == "running", + peer.ID.in(reactor.peers), // InitPeer(peer) and not RemovePeer(peer) + // Routines might not be started, namely: not AddPeer(peer) + // Routines could also be already stopped if Peer has erroed. + updateReactorTo(reactor.RemovePeer(peer, reason)) + } + + // Code: Peer type, onReceive := func(chID byte, msgBytes []byte) + action receive(reactor: ReactorState, e: Envelope): bool = all { + reactor.state == "running", + // The message's sender is an active peer + e.Src.ID.in(reactor.peers), + // Reactor is assigned to the message's channel ID + e.ChannelID.in(reactorsByCh.keys()), + reactorsByCh.get(e.ChannelID) == reactor.name, + reactor.GetChannels().exists(c => c.ID == e.ChannelID), + updateReactorTo(reactor.Receive(e)) + } + + // Code: Switch.OnStop() + action stop(reactor: ReactorState): bool = all { + reactor.state == "running", + // Either no peer was added or all peers were removed + reactor.peers.size() == 0, + updateReactorTo(reactor.OnStop()) + } + + // Simulation support + + action init = all { + reactors' = Map(), + reactorsByCh' = Map(), + } + + // Modelled reactor configuration + pure val reactorName = "myReactor" + pure val reactorChannels = Set({ID: "3", Priority: 1}, {ID: "7", Priority: 2}) + + // For retro-compatibility: the state of the modelled reactor + def state(): ReactorState = { + reactors.get(reactorName) + } + + pure val samplePeers = Set({ID: "p1"}, {ID: "p3"}) + pure val sampleChIDs = Set("1", "3", "7") // ChannelID 1 not registered + pure val sampleMsgs = Set("ping", "pong") + + action step = any { + register(reactorName, NewReactor.with("channels", reactorChannels)), + val reactor = reactors.get(reactorName) + any { + reactor.start(), + reactor.stop(), + nondet peer = oneOf(samplePeers) + any { + // Peer-specific actions + reactor.initPeer(peer), + reactor.addPeer(peer), + reactor.removePeer(peer, "no reason"), + reactor.receive({Src: peer, + ChannelID: oneOf(sampleChIDs), + Message: oneOf(sampleMsgs)}), + } + } + } + +} diff --git a/cometbft/v0.38/spec/rpc/Rpc-Spe.mdx b/cometbft/v0.38/spec/rpc/Rpc-Spe.mdx new file mode 100644 index 00000000..ff9ce2ce --- /dev/null +++ b/cometbft/v0.38/spec/rpc/Rpc-Spe.mdx @@ -0,0 +1,1264 @@ +--- +order: 1 +parent: + title: RPC + order: 6 +--- + +# RPC spec + +This file defines the JSON-RPC spec of CometBFT. This is meant to be implemented by all clients. + +## Support + + | | [CometBFT](https://github.com/cometbft/cometbft/) | [Tendermint-Rs](https://github.com/informalsystems/tendermint-rs) | + |--------------|:----------------------------------------------------------:|:----------------------------------------------------------------:| + | JSON-RPC 2.0 | ✅ | ✅ | + | HTTP | ✅ | ✅ | + | HTTPS | ✅ | ❌ | + | WS | ✅ | ✅ | + + | Routes | [CometBFT](https://github.com/cometbft/cometbft/) | [Tendermint-Rs](https://github.com/informalsystems/tendermint-rs) | + |-----------------------------------------|:----------------------------------------------------------:|:-----------------------------------------------------------------:| + | [Health](#health) | ✅ | ✅ | + | [Status](#status) | ✅ | ✅ | + | [NetInfo](#netinfo) | ✅ | ✅ | + | [Blockchain](#blockchain) | ✅ | ✅ | + | [Block](#block) | ✅ | ✅ | + | [BlockByHash](#blockbyhash) | ✅ | ❌ | + | [BlockResults](#blockresults) | ✅ | ✅ | + | [Commit](#commit) | ✅ | ✅ | + | [Validators](#validators) | ✅ | ✅ | + | [Genesis](#genesis) | ✅ | ✅ | + | [GenesisChunked](#genesischunked) | ✅ | ❌ | + | [ConsensusParams](#consensusparams) | ✅ | ❌ | + | [UnconfirmedTxs](#unconfirmedtxs) | ✅ | ❌ | + | [NumUnconfirmedTxs](#numunconfirmedtxs) | ✅ | ❌ | + | [Tx](#tx) | ✅ | ❌ | + | [BroadCastTxSync](#broadcasttxsync) | ✅ | ✅ | + | [BroadCastTxAsync](#broadcasttxasync) | ✅ | ✅ | + | [ABCIInfo](#abciinfo) | ✅ | ✅ | + | [ABCIQuery](#abciquery) | ✅ | ✅ | + | [BroadcastTxAsync](#broadcasttxasync) | ✅ | ✅ | + | [BroadcastEvidence](#broadcastevidence) | ✅ | ✅ | + +## Timestamps + +Timestamps in the RPC layer of CometBFT follows RFC3339Nano. The RFC3339Nano format removes trailing zeros from the seconds field. + +This means if a block has a timestamp like: `1985-04-12T23:20:50.5200000Z`, the value returned in the RPC will be `1985-04-12T23:20:50.52Z`. + + + +## Info Routes + +### Health + +Node heartbeat + +#### Parameters + +None + +#### Request + +##### HTTP + +```sh +curl http://127.0.0.1:26657/health +``` + +##### JSONRPC + +```sh +curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"health\"}" +``` + +#### Response + +```json +{ + "jsonrpc": "2.0", + "id": -1, + "result": {} +} +``` + +### Status + +Get CometBFT status including node info, pubkey, latest block hash, app hash, block height and time. + +#### Parameters + +None + +#### Request + +##### HTTP + +```sh +curl http://127.0.0.1:26657/status +``` + +##### JSONRPC + +```sh +curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"status\"}" +``` + +#### Response + +```json +{ + "jsonrpc": "2.0", + "id": -1, + "result": { + "node_info": { + "protocol_version": { + "p2p": "8", + "block": "11", + "app": "0" + }, + "id": "b93270b358a72a2db30089f3856475bb1f918d6d", + "listen_addr": "tcp://0.0.0.0:26656", + "network": "cosmoshub-4", + "version": "v0.34.8", + "channels": "40202122233038606100", + "moniker": "aib-hub-node", + "other": { + "tx_index": "on", + "rpc_address": "tcp://0.0.0.0:26657" + } + }, + "sync_info": { + "latest_block_hash": "50F03C0EAACA8BCA7F9C14189ACE9C05A9A1BBB5268DB63DC6A3C848D1ECFD27", + "latest_app_hash": "2316CFF7644219F4F15BEE456435F280E2B38955EEA6D4617CCB6D7ABF781C22", + "latest_block_height": "5622165", + "latest_block_time": "2021-03-25T14:00:43.356134226Z", + "earliest_block_hash": "1455A0C15AC49BB506992EC85A3CD4D32367E53A087689815E01A524231C3ADF", + "earliest_app_hash": "E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855", + "earliest_block_height": "5200791", + "earliest_block_time": "2019-12-11T16:11:34Z", + "catching_up": false + }, + "validator_info": { + "address": "38FB765D0092470989360ECA1C89CD06C2C1583C", + "pub_key": { + "type": "tendermint/PubKeyEd25519", + "value": "Z+8kntVegi1sQiWLYwFSVLNWqdAUGEy7lskL78gxLZI=" + }, + "voting_power": "0" + } + } +} +``` + +### NetInfo + +Network information + +#### Parameters + +None + +#### Request + +##### HTTP + +```sh +curl http://127.0.0.1:26657/net_info +``` + +##### JSONRPC + +```sh +curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"net_info\"}" +``` + +#### Response + +```json +{ + "id": 0, + "jsonrpc": "2.0", + "result": { + "listening": true, + "listeners": [ + "Listener(@)" + ], + "n_peers": "1", + "peers": [ + { + "node_id": "5576458aef205977e18fd50b274e9b5d9014525a", + "url": "tcp://5576458aef205977e18fd50b274e9b5d9014525a@95.179.155.35:26656" + } + ] + } +} +``` + +### Blockchain + +Get block headers. Returned in descending order. May be limited in quantity. + +#### Parameters + +- `minHeight (integer)`: The lowest block to be returned in the response +- `maxHeight (integer)`: The highest block to be returned in the response + +#### Request + +##### HTTP + +```sh +curl http://127.0.0.1:26657/blockchain + +curl http://127.0.0.1:26657/blockchain?minHeight=1&maxHeight=2 +``` + +##### JSONRPC + +```sh +curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"blockchain\",\"params\":{\"minHeight\":\"1\", \"maxHeight\":\"2\"}}" +``` + +#### Response + +```json +{ + "id": 0, + "jsonrpc": "2.0", + "result": { + "last_height": "1276718", + "block_metas": [ + { + "block_id": { + "hash": "112BC173FD838FB68EB43476816CD7B4C6661B6884A9E357B417EE957E1CF8F7", + "parts": { + "total": 1, + "hash": "38D4B26B5B725C4F13571EFE022C030390E4C33C8CF6F88EDD142EA769642DBD" + } + }, + "block_size": 1000000, + "header": { + "version": { + "block": "10", + "app": "0" + }, + "chain_id": "cosmoshub-2", + "height": "12", + "time": "2019-04-22T17:01:51.701356223Z", + "last_block_id": { + "hash": "112BC173FD838FB68EB43476816CD7B4C6661B6884A9E357B417EE957E1CF8F7", + "parts": { + "total": 1, + "hash": "38D4B26B5B725C4F13571EFE022C030390E4C33C8CF6F88EDD142EA769642DBD" + } + }, + "last_commit_hash": "21B9BC845AD2CB2C4193CDD17BFC506F1EBE5A7402E84AD96E64171287A34812", + "data_hash": "970886F99E77ED0D60DA8FCE0447C2676E59F2F77302B0C4AA10E1D02F18EF73", + "validators_hash": "D658BFD100CA8025CFD3BECFE86194322731D387286FBD26E059115FD5F2BCA0", + "next_validators_hash": "D658BFD100CA8025CFD3BECFE86194322731D387286FBD26E059115FD5F2BCA0", + "consensus_hash": "0F2908883A105C793B74495EB7D6DF2EEA479ED7FC9349206A65CB0F9987A0B8", + "app_hash": "223BF64D4A01074DC523A80E76B9BBC786C791FB0A1893AC5B14866356FCFD6C", + "last_results_hash": "", + "evidence_hash": "", + "proposer_address": "D540AB022088612AC74B287D076DBFBC4A377A2E" + }, + "num_txs": "54" + } + ] + } +} +``` + +### Block + +Get block at a specified height. + +#### Parameters + +- `height (integer)`: height of the requested block. If no height is specified the latest block will be used. + +#### Request + +##### HTTP + +```sh +curl http://127.0.0.1:26657/block + +curl http://127.0.0.1:26657/block?height=1 +``` + +##### JSONRPC + +```sh +curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"block\",\"params\":{\"height\":\"1\"}}" +``` + +#### Response + +```json +{ + "id": 0, + "jsonrpc": "2.0", + "result": { + "block_id": { + "hash": "112BC173FD838FB68EB43476816CD7B4C6661B6884A9E357B417EE957E1CF8F7", + "parts": { + "total": 1, + "hash": "38D4B26B5B725C4F13571EFE022C030390E4C33C8CF6F88EDD142EA769642DBD" + } + }, + "block": { + "header": { + "version": { + "block": "10", + "app": "0" + }, + "chain_id": "cosmoshub-2", + "height": "12", + "time": "2019-04-22T17:01:51.701356223Z", + "last_block_id": { + "hash": "112BC173FD838FB68EB43476816CD7B4C6661B6884A9E357B417EE957E1CF8F7", + "parts": { + "total": 1, + "hash": "38D4B26B5B725C4F13571EFE022C030390E4C33C8CF6F88EDD142EA769642DBD" + } + }, + "last_commit_hash": "21B9BC845AD2CB2C4193CDD17BFC506F1EBE5A7402E84AD96E64171287A34812", + "data_hash": "970886F99E77ED0D60DA8FCE0447C2676E59F2F77302B0C4AA10E1D02F18EF73", + "validators_hash": "D658BFD100CA8025CFD3BECFE86194322731D387286FBD26E059115FD5F2BCA0", + "next_validators_hash": "D658BFD100CA8025CFD3BECFE86194322731D387286FBD26E059115FD5F2BCA0", + "consensus_hash": "0F2908883A105C793B74495EB7D6DF2EEA479ED7FC9349206A65CB0F9987A0B8", + "app_hash": "223BF64D4A01074DC523A80E76B9BBC786C791FB0A1893AC5B14866356FCFD6C", + "last_results_hash": "", + "evidence_hash": "", + "proposer_address": "D540AB022088612AC74B287D076DBFBC4A377A2E" + }, + "data": [ + "yQHwYl3uCkKoo2GaChRnd+THLQ2RM87nEZrE19910Z28ABIUWW/t8AtIMwcyU0sT32RcMDI9GF0aEAoFdWF0b20SBzEwMDAwMDASEwoNCgV1YXRvbRIEMzEwMRCd8gEaagom61rphyEDoJPxlcjRoNDtZ9xMdvs+lRzFaHe2dl2P5R2yVCWrsHISQKkqX5H1zXAIJuC57yw0Yb03Fwy75VRip0ZBtLiYsUqkOsPUoQZAhDNP+6LY+RUwz/nVzedkF0S29NZ32QXdGv0=" + ], + "evidence": [ + { + "type": "string", + "height": 0, + "time": 0, + "total_voting_power": 0, + "validator": { + "pub_key": { + "type": "tendermint/PubKeyEd25519", + "value": "A6DoBUypNtUAyEHWtQ9bFjfNg8Bo9CrnkUGl6k6OHN4=" + }, + "voting_power": 0, + "address": "string" + } + } + ], + "last_commit": { + "height": 0, + "round": 0, + "block_id": { + "hash": "112BC173FD838FB68EB43476816CD7B4C6661B6884A9E357B417EE957E1CF8F7", + "parts": { + "total": 1, + "hash": "38D4B26B5B725C4F13571EFE022C030390E4C33C8CF6F88EDD142EA769642DBD" + } + }, + "signatures": [ + { + "type": 2, + "height": "1262085", + "round": 0, + "block_id": { + "hash": "112BC173FD838FB68EB43476816CD7B4C6661B6884A9E357B417EE957E1CF8F7", + "parts": { + "total": 1, + "hash": "38D4B26B5B725C4F13571EFE022C030390E4C33C8CF6F88EDD142EA769642DBD" + } + }, + "timestamp": "2019-08-01T11:39:38.867269833Z", + "validator_address": "000001E443FD237E4B616E2FA69DF4EE3D49A94F", + "validator_index": 0, + "signature": "DBchvucTzAUEJnGYpNvMdqLhBAHG4Px8BsOBB3J3mAFCLGeuG7uJqy+nVngKzZdPhPi8RhmE/xcw/M9DOJjEDg==" + } + ] + } + } + } +} +``` + +### BlockByHash + +#### Parameters + +- `hash (string)`: Hash of the block to query for. + +#### Request + +##### HTTP + +```sh +curl http://127.0.0.1:26657/block_by_hash?hash=0xD70952032620CC4E2737EB8AC379806359D8E0B17B0488F627997A0B043ABDED +``` + +##### JSONRPC + +```sh +curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"block_by_hash\",\"params\":{\"hash\":\"0xD70952032620CC4E2737EB8AC379806359D8E0B17B0488F627997A0B043ABDED\"}}" +``` + +#### Response + +```json +{ + "id": 0, + "jsonrpc": "2.0", + "result": { + "block_id": { + "hash": "112BC173FD838FB68EB43476816CD7B4C6661B6884A9E357B417EE957E1CF8F7", + "parts": { + "total": 1, + "hash": "38D4B26B5B725C4F13571EFE022C030390E4C33C8CF6F88EDD142EA769642DBD" + } + }, + "block": { + "header": { + "version": { + "block": "10", + "app": "0" + }, + "chain_id": "cosmoshub-2", + "height": "12", + "time": "2019-04-22T17:01:51.701356223Z", + "last_block_id": { + "hash": "112BC173FD838FB68EB43476816CD7B4C6661B6884A9E357B417EE957E1CF8F7", + "parts": { + "total": 1, + "hash": "38D4B26B5B725C4F13571EFE022C030390E4C33C8CF6F88EDD142EA769642DBD" + } + }, + "last_commit_hash": "21B9BC845AD2CB2C4193CDD17BFC506F1EBE5A7402E84AD96E64171287A34812", + "data_hash": "970886F99E77ED0D60DA8FCE0447C2676E59F2F77302B0C4AA10E1D02F18EF73", + "validators_hash": "D658BFD100CA8025CFD3BECFE86194322731D387286FBD26E059115FD5F2BCA0", + "next_validators_hash": "D658BFD100CA8025CFD3BECFE86194322731D387286FBD26E059115FD5F2BCA0", + "consensus_hash": "0F2908883A105C793B74495EB7D6DF2EEA479ED7FC9349206A65CB0F9987A0B8", + "app_hash": "223BF64D4A01074DC523A80E76B9BBC786C791FB0A1893AC5B14866356FCFD6C", + "last_results_hash": "", + "evidence_hash": "", + "proposer_address": "D540AB022088612AC74B287D076DBFBC4A377A2E" + }, + "data": [ + "yQHwYl3uCkKoo2GaChRnd+THLQ2RM87nEZrE19910Z28ABIUWW/t8AtIMwcyU0sT32RcMDI9GF0aEAoFdWF0b20SBzEwMDAwMDASEwoNCgV1YXRvbRIEMzEwMRCd8gEaagom61rphyEDoJPxlcjRoNDtZ9xMdvs+lRzFaHe2dl2P5R2yVCWrsHISQKkqX5H1zXAIJuC57yw0Yb03Fwy75VRip0ZBtLiYsUqkOsPUoQZAhDNP+6LY+RUwz/nVzedkF0S29NZ32QXdGv0=" + ], + "evidence": [ + { + "type": "string", + "height": 0, + "time": 0, + "total_voting_power": 0, + "validator": { + "pub_key": { + "type": "tendermint/PubKeyEd25519", + "value": "A6DoBUypNtUAyEHWtQ9bFjfNg8Bo9CrnkUGl6k6OHN4=" + }, + "voting_power": 0, + "address": "string" + } + } + ], + "last_commit": { + "height": 0, + "round": 0, + "block_id": { + "hash": "112BC173FD838FB68EB43476816CD7B4C6661B6884A9E357B417EE957E1CF8F7", + "parts": { + "total": 1, + "hash": "38D4B26B5B725C4F13571EFE022C030390E4C33C8CF6F88EDD142EA769642DBD" + } + }, + "signatures": [ + { + "type": 2, + "height": "1262085", + "round": 0, + "block_id": { + "hash": "112BC173FD838FB68EB43476816CD7B4C6661B6884A9E357B417EE957E1CF8F7", + "parts": { + "total": 1, + "hash": "38D4B26B5B725C4F13571EFE022C030390E4C33C8CF6F88EDD142EA769642DBD" + } + }, + "timestamp": "2019-08-01T11:39:38.867269833Z", + "validator_address": "000001E443FD237E4B616E2FA69DF4EE3D49A94F", + "validator_index": 0, + "signature": "DBchvucTzAUEJnGYpNvMdqLhBAHG4Px8BsOBB3J3mAFCLGeuG7uJqy+nVngKzZdPhPi8RhmE/xcw/M9DOJjEDg==" + } + ] + } + } + } +} +``` + +### BlockResults + +### Parameters + +- `height (integer)`: Height of the block which contains the results. If no height is specified, the latest block height will be used + +#### Request + +##### HTTP + +```sh +curl http://127.0.0.1:26657/block_results + + +curl http://127.0.0.1:26657/block_results?height=1 +``` + +##### JSONRPC + +```sh +curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"block_results\",\"params\":{\"height\":\"1\"}}" +``` + +#### Response + +```json +{ + "jsonrpc": "2.0", + "id": 0, + "result": { + "height": "12", + "total_gas_used": "100", + "txs_results": [ + { + "code": "0", + "data": "", + "log": "not enough gas", + "info": "", + "gas_wanted": "100", + "gas_used": "100", + "events": [ + { + "type": "app", + "attributes": [ + { + "key": "YWN0aW9u", + "value": "c2VuZA==", + "index": false + } + ] + } + ], + "codespace": "ibc" + } + ], + "begin_block_events": [ + { + "type": "app", + "attributes": [ + { + "key": "YWN0aW9u", + "value": "c2VuZA==", + "index": false + } + ] + } + ], + "end_block": [ + { + "type": "app", + "attributes": [ + { + "key": "YWN0aW9u", + "value": "c2VuZA==", + "index": false + } + ] + } + ], + "validator_updates": [ + { + "pub_key": { + "type": "tendermint/PubKeyEd25519", + "value": "9tK9IT+FPdf2qm+5c2qaxi10sWP+3erWTKgftn2PaQM=" + }, + "power": "300" + } + ], + "consensus_params_updates": { + "block": { + "max_bytes": "22020096", + "max_gas": "1000", + "time_iota_ms": "1000" + }, + "evidence": { + "max_age": "100000" + }, + "validator": { + "pub_key_types": [ + "ed25519" + ] + } + } + } +} +``` + +### Commit + +#### Parameters + +- `height (integer)`: Height of the block the requested commit pertains to. If no height is set the latest commit will be returned. + +#### Request + +##### HTTP + +```sh +curl http://127.0.0.1:26657/commit + + +curl http://127.0.0.1:26657/commit?height=1 +``` + +##### JSONRPC + +```sh +curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"commit\",\"params\":{\"height\":\"1\"}}" +``` + +#### Response + +```json +{ + "jsonrpc": "2.0", + "id": 0, + "result": { + "signed_header": { + "header": { + "version": { + "block": "10", + "app": "0" + }, + "chain_id": "cosmoshub-2", + "height": "12", + "time": "2019-04-22T17:01:51.701356223Z", + "last_block_id": { + "hash": "112BC173FD838FB68EB43476816CD7B4C6661B6884A9E357B417EE957E1CF8F7", + "parts": { + "total": 1, + "hash": "38D4B26B5B725C4F13571EFE022C030390E4C33C8CF6F88EDD142EA769642DBD" + } + }, + "last_commit_hash": "21B9BC845AD2CB2C4193CDD17BFC506F1EBE5A7402E84AD96E64171287A34812", + "data_hash": "970886F99E77ED0D60DA8FCE0447C2676E59F2F77302B0C4AA10E1D02F18EF73", + "validators_hash": "D658BFD100CA8025CFD3BECFE86194322731D387286FBD26E059115FD5F2BCA0", + "next_validators_hash": "D658BFD100CA8025CFD3BECFE86194322731D387286FBD26E059115FD5F2BCA0", + "consensus_hash": "0F2908883A105C793B74495EB7D6DF2EEA479ED7FC9349206A65CB0F9987A0B8", + "app_hash": "223BF64D4A01074DC523A80E76B9BBC786C791FB0A1893AC5B14866356FCFD6C", + "last_results_hash": "", + "evidence_hash": "", + "proposer_address": "D540AB022088612AC74B287D076DBFBC4A377A2E" + }, + "commit": { + "height": "1311801", + "round": 0, + "block_id": { + "hash": "112BC173FD838FB68EB43476816CD7B4C6661B6884A9E357B417EE957E1CF8F7", + "parts": { + "total": 1, + "hash": "38D4B26B5B725C4F13571EFE022C030390E4C33C8CF6F88EDD142EA769642DBD" + } + }, + "signatures": [ + { + "block_id_flag": 2, + "validator_address": "000001E443FD237E4B616E2FA69DF4EE3D49A94F", + "timestamp": "2019-04-22T17:01:58.376629719Z", + "signature": "14jaTQXYRt8kbLKEhdHq7AXycrFImiLuZx50uOjs2+Zv+2i7RTG/jnObD07Jo2ubZ8xd7bNBJMqkgtkd0oQHAw==" + } + ] + } + }, + "canonical": true + } +} +``` + +### Validators + +#### Parameters + +- `height (integer)`: Block height at which the validators were present on. If no height is set the latest commit will be returned. +- `page (integer)`: +- `per_page (integer)`: + +#### Request + +##### HTTP + +```sh +curl http://127.0.0.1:26657/validators +``` + +##### JSONRPC + +```sh +curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"validators\",\"params\":{\"height\":\"1\", \"page\":\"1\", \"per_page\":\"20\"}}" +``` + +#### Response + +```json +{ + "jsonrpc": "2.0", + "id": 0, + "result": { + "block_height": "55", + "validators": [ + { + "address": "000001E443FD237E4B616E2FA69DF4EE3D49A94F", + "pub_key": { + "type": "tendermint/PubKeyEd25519", + "value": "9tK9IT+FPdf2qm+5c2qaxi10sWP+3erWTKgftn2PaQM=" + }, + "voting_power": "239727", + "proposer_priority": "-11896414" + } + ], + "count": "1", + "total": "25" + } +} +``` + +### Genesis + +Get Genesis of the chain. If the response is large, this operation +will return an error: use `genesis_chunked` instead. + +#### Request + +##### HTTP + +```sh +curl http://127.0.0.1:26657/genesis +``` + +##### JSONRPC + +```sh +curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"genesis\"}" +``` + +#### Response + +```json +{ + "jsonrpc": "2.0", + "id": 0, + "result": { + "genesis": { + "genesis_time": "2019-04-22T17:00:00Z", + "chain_id": "cosmoshub-2", + "initial_height": "2", + "consensus_params": { + "block": { + "max_bytes": "22020096", + "max_gas": "1000", + "time_iota_ms": "1000" + }, + "evidence": { + "max_age": "100000" + }, + "validator": { + "pub_key_types": [ + "ed25519" + ] + } + }, + "validators": [ + { + "address": "B00A6323737F321EB0B8D59C6FD497A14B60938A", + "pub_key": { + "type": "tendermint/PubKeyEd25519", + "value": "cOQZvh/h9ZioSeUMZB/1Vy1Xo5x2sjrVjlE/qHnYifM=" + }, + "power": "9328525", + "name": "Certus One" + } + ], + "app_hash": "", + "app_state": {} + } + } +} +``` + +### GenesisChunked + +Get the genesis document in a chunks to support easily transfering larger documents. + +#### Parameters + +- `chunk` (integer): the index number of the chunk that you wish to + fetch. These IDs are 0 indexed. + +#### Request + +##### HTTP + +```sh +curl http://127.0.0.1:26657/genesis_chunked?chunk=0 +``` + +##### JSONRPC + +```sh +curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"genesis_chunked\",\"params\":{\"chunk\":0}}" +``` + +#### Response + +```json +{ + "jsonrpc": "2.0", + "id": 0, + "result": { + "chunk": 0, + "total": 10, + "data": "dGVuZGVybWludAo=" + } +} +``` + +### ConsensusParams + +Get the consensus parameters. + +#### Parameters + +- `height (integer)`: Block height at which the consensus params would like to be fetched for. + +#### Request + +##### HTTP + +```sh +curl http://127.0.0.1:26657/consensus_params +``` + +##### JSONRPC + +```sh +curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"consensus_params\"}" +``` + +#### Response + +```json +{ + "jsonrpc": "2.0", + "id": 0, + "result": { + "block_height": "1", + "consensus_params": { + "block": { + "max_bytes": "22020096", + "max_gas": "1000", + "time_iota_ms": "1000" + }, + "evidence": { + "max_age": "100000" + }, + "validator": { + "pub_key_types": [ + "ed25519" + ] + } + } + } +} +``` + +### UnconfirmedTxs + +Get a list of unconfirmed transactions. + +#### Parameters + +- `limit (integer)` The amount of txs to respond with. + +#### Request + +##### HTTP + +```sh +curl http://127.0.0.1:26657/unconfirmed_txs +``` + +##### JSONRPC + +```sh +curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"unconfirmed_txs\, \"params\":{\"limit\":\"20\"}}" +``` + +#### Response + +```json +{ + "jsonrpc": "2.0", + "id": 0, + "result": { + "n_txs": "82", + "total": "82", + "total_bytes": "19974", + "txs": [ + "gAPwYl3uCjCMTXENChSMnIkb5ZpYHBKIZqecFEV2tuZr7xIUA75/FmYq9WymsOBJ0XSJ8yV8zmQKMIxNcQ0KFIyciRvlmlgcEohmp5wURXa25mvvEhQbrvwbvlNiT+Yjr86G+YQNx7kRVgowjE1xDQoUjJyJG+WaWBwSiGannBRFdrbma+8SFK2m+1oxgILuQLO55n8mWfnbIzyPCjCMTXENChSMnIkb5ZpYHBKIZqecFEV2tuZr7xIUQNGfkmhTNMis4j+dyMDIWXdIPiYKMIxNcQ0KFIyciRvlmlgcEohmp5wURXa25mvvEhS8sL0D0wwgGCItQwVowak5YB38KRIUCg4KBXVhdG9tEgUxMDA1NBDoxRgaagom61rphyECn8x7emhhKdRCB2io7aS/6Cpuq5NbVqbODmqOT3jWw6kSQKUresk+d+Gw0BhjiggTsu8+1voW+VlDCQ1GRYnMaFOHXhyFv7BCLhFWxLxHSAYT8a5XqoMayosZf9mANKdXArA=" + ] + } +} +``` + +### NumUnconfirmedTxs + +Get data about unconfirmed transactions. + +#### Parameters + +None + +#### Request + +##### HTTP + +```sh +curl http://127.0.0.1:26657/num_unconfirmed_txs +``` + +##### JSONRPC + +```sh +curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"num_unconfirmed_txs\"}" +``` + +#### Response + +```json +{ + "jsonrpc": "2.0", + "id": 0, + "result": { + "n_txs": "31", + "total": "82", + "total_bytes": "19974" + } +} +``` + +### Tx + +#### Parameters + +- `hash (string)`: The hash of the transaction +- `prove (bool)`: If the response should include proof the transaction was included in a block. + +#### Request + +##### HTTP + +```sh +curl http://127.0.0.1:26657/num_unconfirmed_txs +``` + +##### JSONRPC + +```sh +curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"num_unconfirmed_txs\"}" +``` + +#### Response + +```json +{ + "jsonrpc": "2.0", + "id": 0, + "result": { + "hash": "D70952032620CC4E2737EB8AC379806359D8E0B17B0488F627997A0B043ABDED", + "height": "1000", + "index": 0, + "tx_result": { + "log": "[{\"msg_index\":\"0\",\"success\":true,\"log\":\"\"}]", + "gas_wanted": "200000", + "gas_used": "28596", + "tags": [ + { + "key": "YWN0aW9u", + "value": "c2VuZA==", + "index": false + } + ] + }, + "tx": "5wHwYl3uCkaoo2GaChQmSIu8hxpJxLcCuIi8fiHN4TMwrRIU/Af1cEG7Rcs/6LjTl7YjRSymJfYaFAoFdWF0b20SCzE0OTk5OTk1MDAwEhMKDQoFdWF0b20SBDUwMDAQwJoMGmoKJuta6YchAwswBShaB1wkZBctLIhYqBC3JrAI28XGzxP+rVEticGEEkAc+khTkKL9CDE47aDvjEHvUNt+izJfT4KVF2v2JkC+bmlH9K08q3PqHeMI9Z5up+XMusnTqlP985KF+SI5J3ZOIhhNYWRlIGJ5IENpcmNsZSB3aXRoIGxvdmU=" + } +} +``` + +## Transaction Routes + +### BroadCastTxSync + +Returns with the response from CheckTx. Does not wait for DeliverTx result. + +#### Parameters + +- `tx (string)`: The transaction encoded + +#### Request + +##### HTTP + +```sh +curl http://127.0.0.1:26657/broadcast_tx_sync?tx=encoded_tx +``` + +##### JSONRPC + +```sh +curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"broadcast_tx_sync\",\"params\":{\"tx\":\"a/encoded_tx/c\"}}" +``` + +#### Response + +```json +{ + "jsonrpc": "2.0", + "id": 0, + "result": { + "code": "0", + "data": "", + "log": "", + "codespace": "ibc", + "hash": "0D33F2F03A5234F38706E43004489E061AC40A2E" + }, + "error": "" +} +``` + +### BroadCastTxAsync + +Returns right away, with no response. Does not wait for CheckTx nor DeliverTx results. + +#### Parameters + +- `tx (string)`: The transaction encoded + +#### Request + +##### HTTP + +```sh +curl http://127.0.0.1:26657/broadcast_tx_async?tx=encoded_tx +``` + +##### JSONRPC + +```sh +curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"broadcast_tx_async\",\"params\":{\"tx\":\"a/encoded_tx/c\"}}" +``` + +#### Response + +```json +{ + "jsonrpc": "2.0", + "id": 0, + "result": { + "code": "0", + "data": "", + "log": "", + "codespace": "ibc", + "hash": "0D33F2F03A5234F38706E43004489E061AC40A2E" + }, + "error": "" +} +``` + +### CheckTx + +Checks the transaction without executing it. + +#### Parameters + +- `tx (string)`: String of the encoded transaction + +#### Request + +##### HTTP + +```sh +curl http://127.0.0.1:26657/check_tx?tx=encoded_tx +``` + +##### JSONRPC + +```sh +curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"check_tx\",\"params\":{\"tx\":\"a/encoded_tx/c\"}}" +``` + +#### Response + +```json +{ + "id": 0, + "jsonrpc": "2.0", + "error": "", + "result": { + "code": "0", + "data": "", + "log": "", + "info": "", + "gas_wanted": "1", + "gas_used": "0", + "events": [ + { + "type": "app", + "attributes": [ + { + "key": "YWN0aW9u", + "value": "c2VuZA==", + "index": false + } + ] + } + ], + "codespace": "bank" + } +} +``` + +## ABCI Routes + +### ABCIInfo + +Get some info about the application. + +#### Parameters + +None + +#### Request + +##### HTTP + +```sh +curl http://127.0.0.1:26657/abci_info +``` + +##### JSONRPC + +```sh +curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"abci_info\"}" +``` + +#### Response + +```json +{ + "jsonrpc": "2.0", + "id": 0, + "result": { + "response": { + "data": "{\"size\":0}", + "version": "0.16.1", + "app_version": "1314126" + } + } +} +``` + +### ABCIQuery + +Query the application for some information. + +#### Parameters + +- `path (string)`: Path to the data. This is defined by the application. +- `data (string)`: The data requested +- `height (integer)`: Height at which the data is being requested for. +- `prove (bool)`: Include proofs of the transactions inclusion in the block + +#### Request + +##### HTTP + +```sh +curl http://127.0.0.1:26657/abci_query?path="a/b/c"=IHAVENOIDEA&height=1&prove=true +``` + +##### JSONRPC + +```sh +curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"abci_query\",\"params\":{\"path\":\"a/b/c\", \"height\":\"1\", \"bool\":\"true\"}}" +``` + +#### Response + +```json +{ + "error": "", + "result": { + "response": { + "log": "exists", + "height": "0", + "proof": "010114FED0DAD959F36091AD761C922ABA3CBF1D8349990101020103011406AA2262E2F448242DF2C2607C3CDC705313EE3B0001149D16177BC71E445476174622EA559715C293740C", + "value": "61626364", + "key": "61626364", + "index": "-1", + "code": "0" + } + }, + "id": 0, + "jsonrpc": "2.0" +} +``` + +## Evidence Routes + +### BroadcastEvidence + +Broadcast evidence of the misbehavior. + +#### Parameters + +- `evidence (string)`: + +#### Request + +##### HTTP + +```sh +curl http://localhost:26657/broadcast_evidence?evidence=JSON_EVIDENCE_encoded +``` + +#### JSONRPC + +```sh +curl -X POST https://localhost:26657 -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"broadcast_evidence\",\"params\":{\"evidence\":\"JSON_EVIDENCE_encoded\"}}" +``` + +#### Response + +```json +{ + "error": "", + "result": "", + "id": 0, + "jsonrpc": "2.0" +} +``` diff --git a/docs.json b/docs.json index b0a1747c..86bcccf7 100644 --- a/docs.json +++ b/docs.json @@ -123,6 +123,219 @@ "pages": [ "index" ] + }, + { + "dropdown": "CometBFT", + "versions": [ + { + "version": "v0.38", + "tabs": [ + { + "tab": "Learn", + "groups": [ + { + "group": "CometBFT", + "pages": [ + "cometbft/v0.38/docs/README", + "cometbft/v0.38/docs/introduction/intro" + ] + }, + { + "group": "Guides", + "pages": [ + "cometbft/v0.38/docs/guides/Quick-Start", + "cometbft/v0.38/docs/guides/Install-CometBFT", + "cometbft/v0.38/docs/guides/Creating-an-application-in-Go", + "cometbft/v0.38/docs/guides/Creating-a-built-in-application-in-Go" + ] + }, + { + "group": "Apps", + "pages": [ + "cometbft/v0.38/docs/app-dev/Getting-Started", + "cometbft/v0.38/docs/app-dev/Application-Architecture-Guide", + "cometbft/v0.38/docs/app-dev/Using-ABCI-CLI", + "cometbft/v0.38/docs/app-dev/Indexing-Transactions" + ] + }, + { + "group": "Core", + "pages": [ + "cometbft/v0.38/docs/core/Using-CometBFT", + "cometbft/v0.38/docs/core/Running-in-production", + "cometbft/v0.38/docs/core/configuration", + "cometbft/v0.38/docs/core/mempool", + "cometbft/v0.38/docs/core/block-sync", + "cometbft/v0.38/docs/core/state-sync", + "cometbft/v0.38/docs/core/RPC", + "cometbft/v0.38/docs/core/Subscribing-to-events-via-Websocket", + "cometbft/v0.38/docs/core/metrics", + "cometbft/v0.38/docs/core/Validators", + "cometbft/v0.38/docs/core/light-client", + "cometbft/v0.38/docs/core/block-structure", + "cometbft/v0.38/docs/core/how-to-read-logs" + ] + }, + { + "group": "Tools", + "pages": [ + "cometbft/v0.38/docs/tools/Overview", + "cometbft/v0.38/docs/tools/debugging" + ] + }, + { + "group": "Networks", + "pages": [ + "cometbft/v0.38/docs/networks/Overview", + "cometbft/v0.38/docs/networks/Docker-Compose" + ] + }, + { + "group": "CometBFT Quality Assurance", + "pages": [ + "cometbft/v0.38/docs/qa/CometBFT-QA", + "cometbft/v0.38/docs/qa/Method", + "cometbft/v0.38/docs/qa/CometBFT-QA-38", + "cometbft/v0.38/docs/qa/CometBFT-QA-37", + "cometbft/v0.38/docs/qa/CometBFT-QA-34", + "cometbft/v0.38/docs/qa/TMCore-QA-37", + "cometbft/v0.38/docs/qa/TMCore-QA-34" + ] + }, + { + "group": "Architecture & ADRs", + "pages": [ + "cometbft/v0.38/docs/architecture/README", + "cometbft/v0.38/docs/architecture/adr-111-nop-mempool", + "cometbft/v0.38/docs/architecture/adr-template" + ] + }, + { + "group": "RFCs", + "pages": [ + "cometbft/v0.38/docs/rfc/README", + "cometbft/v0.38/docs/rfc/rfc-100-abci-vote-extension-propag", + "cometbft/v0.38/docs/rfc/rfc-template" + ] + } + ] + }, + { + "tab": "Specification", + "groups": [ + { + "group": "CometBFT Spec", + "pages": [ + "cometbft/v0.38/spec/CometBFT-Spec" + ] + }, + { + "group": "Core", + "pages": [ + "cometbft/v0.38/spec/core/Overview", + "cometbft/v0.38/spec/core/Data_structures", + "cometbft/v0.38/spec/core/encoding", + "cometbft/v0.38/spec/core/genesis", + "cometbft/v0.38/spec/core/state" + ] + }, + { + "group": "ABCI++", + "pages": [ + "cometbft/v0.38/spec/abci/Overview", + "cometbft/v0.38/spec/abci/Outline", + "cometbft/v0.38/spec/abci/Methods", + "cometbft/v0.38/spec/abci/Requirements-for-the-Application", + "cometbft/v0.38/spec/abci/CometBFTs-expected-behavior", + "cometbft/v0.38/spec/abci/Client-and-server", + "cometbft/v0.38/spec/abci/Introduction" + ] + }, + { + "group": "Consensus", + "pages": [ + "cometbft/v0.38/spec/consensus/Overview", + "cometbft/v0.38/spec/consensus/Consensus-Paper", + "cometbft/v0.38/spec/consensus/Byzantine-Consensus-Algorithm", + "cometbft/v0.38/spec/consensus/Light-Client", + "cometbft/v0.38/spec/consensus/Creating-Proposal", + "cometbft/v0.38/spec/consensus/BFT-Time", + "cometbft/v0.38/spec/consensus/Proposer-Selection", + "cometbft/v0.38/spec/consensus/Evidence", + "cometbft/v0.38/spec/consensus/Validator-Signing", + "cometbft/v0.38/spec/consensus/WAL" + ] + }, + { + "group": "Light Client", + "pages": [ + "cometbft/v0.38/spec/light-client/Light-Client-Specification", + "cometbft/v0.38/spec/light-client/verification", + "cometbft/v0.38/spec/light-client/Fork-Detection", + "cometbft/v0.38/spec/light-client/Accountability" + ] + }, + { + "group": "P2P", + "pages": [ + "cometbft/v0.38/spec/p2p/Peer-to-Peer", + "cometbft/v0.38/spec/p2p/Implementation-of-the-p2p-layer", + { + "group": "Legacy Docs", + "pages": [ + "cometbft/v0.38/spec/p2p/legacy-docs/Overview", + { + "group": "Messages", + "pages": [ + "cometbft/v0.38/spec/p2p/legacy-docs/messages/Overview", + "cometbft/v0.38/spec/p2p/legacy-docs/messages/block-sync", + "cometbft/v0.38/spec/p2p/legacy-docs/messages/evidence", + "cometbft/v0.38/spec/p2p/legacy-docs/messages/mempool", + "cometbft/v0.38/spec/p2p/legacy-docs/messages/state-sync", + "cometbft/v0.38/spec/p2p/legacy-docs/messages/Peer-Exchange", + "cometbft/v0.38/spec/p2p/legacy-docs/messages/consensus" + ] + }, + "cometbft/v0.38/spec/p2p/legacy-docs/P2P-Multiplex-Connection", + "cometbft/v0.38/spec/p2p/legacy-docs/Peers", + "cometbft/v0.38/spec/p2p/legacy-docs/P2P-Config", + "cometbft/v0.38/spec/p2p/legacy-docs/Peer-Discovery", + { + "group": "Reactors", + "pages": [ + "cometbft/v0.38/spec/p2p/reactor-api/Reactors", + "cometbft/v0.38/spec/p2p/reactor-api/Reactor-Api", + "cometbft/v0.38/spec/p2p/reactor-api/API-for-Reactors" + ] + } + ] + } + ] + }, + { + "group": "RPC", + "pages": [ + "cometbft/v0.38/spec/rpc/Rpc-Spe" + ] + }, + { + "group": "Blockchain", + "pages": [ + "cometbft/v0.38/spec/blockchain/Blockchain" + ] + }, + { + "group": "Ivy Proofs", + "pages": [ + "cometbft/v0.38/spec/ivy-proofs/README" + ] + } + ] + } + ] + } + ], + "icon": "/assets/icons/sdk.svg" }, { "dropdown": "Cosmos SDK", diff --git a/versions.json b/versions.json index 6ff8c271..a4e11882 100644 --- a/versions.json +++ b/versions.json @@ -35,6 +35,14 @@ "defaultVersion": "next", "repository": "cosmos/ibc-go", "changelogPath": "CHANGELOG.md" + }, + "cometbft": { + "versions": [ + "v0.38" + ], + "defaultVersion": "v0.38", + "repository": "cometbft/cometbft", + "changelogPath": "CHANGELOG.md" } } }