Conversation
sydneynotthecity
left a comment
sydneynotthecity
left a comment
There was a problem hiding this comment.
Couple of questions regarding the newly added elements in the Output structs, otherwise looks good!
cmd/export_ledger_entry_changes.go
Outdated
| } | ||
|
|
||
| for checkpointLedgers := range verifyOutputs { | ||
| _, err := verify.VerifyState(ctx, verifyOutputs[checkpointLedgers], archive, checkpointLedgers, verifyBatchSize) |
There was a problem hiding this comment.
Do you know where the verifyBatchSize comes from? Do we know that 50000 is big enough?
There was a problem hiding this comment.
This value can be changed. It's basically as big as possible but small enough to not cause OOM crash.
internal/utils/verify/verify.go
Outdated
| return false, errors.Wrap(err, "addLiquidityPoolsToStateVerifier failed") | ||
| } | ||
|
|
||
| return true, nil |
There was a problem hiding this comment.
There should be Verify() call after Write()'ing all ledger entries.
There was a problem hiding this comment.
@bartekn calling Verify() against StateVerifier.currentEntries makes sense when these entries come from the Captive Core? I'm struggling to implement this method as it is because the logic I used in this file is different than the original one.
If implementing this is really necessary, could you shed some light on how to?
There was a problem hiding this comment.
This method needs one argument which is the total number of ledger entries in the user's storage (in our case: BigQuery). This is necessary because due to a bug users can have extra entries. If the provided number is equal to the total number streamed state is correct, if not it means you have extra entries. Obviously you can fool this by passing the number of entries you got from state verifier but it's better to count all entries in the storage.
sydneynotthecity
left a comment
sydneynotthecity
left a comment
There was a problem hiding this comment.
Looks good and ready for release
3 participants
Overview
This implements Horizon's
verifypackage tostellar-etlcommandexport_ledger_entry_changes. It only verifies entries from checkpoint ledgers (every 64 ledgers, every ~5 minutes).Changes
Raw*fields typesAccountOutput,OfferOutput,PoolOutput,ClaimableBalanceOutputandTrustlineOutputLedgerIsCheckpoint()function toutilspackageTransformedOutputtype tointernal/transform/schema.goto assist the Horizon'sverifypackage implementationinternal/utils/verify.goto work withstellar-etlCloses GH#279
Closes GH#280
Closes GH#281