blockchain: scalable state proposal

[oleganza]

Per discussion in #230, here's a concrete proposal for addressing scalability of the blockchain state:

• Core node contains "trimmed state" of log₂(utxos) size via utreexo protocol. If it has its own utxos, it needs to track updates to the state and adjust its merkle proofs for its utxos. This means, that all devices w/o bandwidth constraints have low storage overhead and can use core nodes for all their blockchain needs (wallets, issuers, exchanges, point-of-sale terminals, payment processors etc).
• Thick node encapsulates Core node and stores the full UTXO state and allows non-nodes to simply keep their utxos: then it will provide up-to-date proofs or insert them on the fly.
• Core nodes can maintain proofs for N most recent utxos so that other nodes can avoid sending the proofs for them. This dramatically improves bandwidth as nodes only need to transmit proofs for older UTXO that are pruned from such limited buffer. N can be chosen on per-connection basis.
• There are no nonces to store. For non-replayability, every transaction must spend a utxo.
• Instruction nonce can be removed.
• To enable issuance of custom assets, blocks mint synthetic nonce-units abbreviated as nits. These can be given away to anyone who needs to issue assets, so they can make their txs non-replayable.
• Nits are minted at a pre-determinate and capped schedule in order to avoid u64 overflow like for any other assets. Every block is permitted to add an artificial nits-containing utxo to the state with arbitrary predicate, pre-determinate quantity, zero flavor and anchor set to the previous block ID.
• Minted utxos are remembered with a 48h expiration time in a separate maturity buffer and checked for attempted spending before the maturity time. This is to incentivize placing transactions in a continuous chain of blocks.
• SPV clients can subscribe to Core nodes to track utxo proofs for them, and receive up-to-date proofs on continuous basis, so they can (1) detect misbehavior early and switch to a different Core node; (2) watch payment channel status and detect forced settlements to be able to contest invalid settlements.

[vickiniu]

Just to clarify, each core node will track:

• maturity buffer of UTXOs from last 48h
• N most recent utxos
• proofs for all of that nodes' UTXOs

Maybe a nit, but it seems like it might be easier to have UTXOs committed to the UTreeXO struct every x number of blocks (rather than the 48h rolling window?), or I might be misunderstanding the maturity buffer.

[oleganza]

Only nonce-minting utxos have to sit in the maturity buffer to prevent their spending until 48h in the future. This has nothing to do with remembering N most recent utxos (per each node's RAM availability) which helps avoiding transmitting the utreexo proofs.

Separate topic, whether utreexo has to be recomputed every M blocks (or every M utxos), or on every block. This may help with performance:

1. all utxos that come-and-go soon will have lower impact on re-hashing of the tree — since they'll get born and die in-between recalculations.
2. spv clients who have to watch for their utxo proof updates only have to follow every some number of blocks, and save bandwidth.
3. prior mentioned N has now lower bound set to M - meaning, every node guarantees that there's some buffer of proofs, so those don't need to be transmitted.

Problems with this choice:

1. It makes more sense to cap remembered utxos by their number, not by time or block count. But this means, the interval between blocks is variable, so spv clients need some explicit reference to "previous block with commitment is X" in the block header.
2. this cap has to be synced by all parties - maybe we need to precommit to it in the block header and allow changing it for future commitment point.

[oleganza]

On representation of minted utxos:

1. To save space, the block commits to a list of tuples: (qty, predicate), from which it creates Output structures: anchor as H(prevblockid, index), flavor commitment: identity point, qty commitment computed on the fly from qty and compressed.
2. All quantities are summed up and checked for being ≤ than allowed amount for this block.
3. UTXOs are added to the utxo set before applying spent/added utxos in transactions.
4. UTXOs are also remembered with +48h expiration time in a separate maturity buffer. Every spent utxo is checked to not be in this buffer. Note: this buffer is more efficient than embedding maxtime checks into the predicate, but less scalable than it — so it's reserved only for minted nonces.

[oleganza]

Update:

Instead of special-cased nits, we can have a rule that initial block is tied to a utxo set that's pre-defined (which means the utxo set is height=0, and initial block has height=1). Initial UTXO set then can define something like nits, or any arbitrary asset possibly representing a "share in the network" or whatever.

[oleganza]

Nonce units are being developed here: #317