Allow specification of classic transaction sizes in blended loadgen

[bboston7]

The blended loadgen mode uses the PAY loadgen mode as the only source of classic transactions. However, it would be useful to be able to specify a distribution a sizes for these payment transactions, ideally in bytes rather than number of operations.

[drebelsky]

Currently, we have config variables LOADGEN_OP_COUNT_FOR_TESTING and LOADGEN_OP_COUNT_DISTRIBUTION_FOR_TESTING, which are used for pretend transactions in LoadGenMode::PRETEND and payment transactions in LoadGenMode::MIXED_CLASSIC. If we're going to use a byte distribution for LoadGenMode::MIXED_CLASSIC_SOROBAN, we should probably use it for these two, as well. Conversely, if we keep the op count distribution, we should probably use that for MIXED_CLASSIC_SOROBAN also. For comparison, the Soroban side of MIXED_CLASSIC_SOROBAN (and SOROBAN_INVOKE) use distributions for wasm bytes, num data entries, io kilobytes, tx size bytes, and instructions.
@marta-lokhova

[bboston7]

If we're going to use a byte distribution for LoadGenMode::MIXED_CLASSIC_SOROBAN, we should probably use it for these two

My vote is for this. It's easier to think in terms of bytes than ops, especially given that we're primarily stressing the overlay with these tests. I'm curious what @marta-lokhova thinks though.

[drebelsky]

Oops, forgot to update this. Chatted in person: the current plan is to switch to bytes everywhere so that it's more clear what loadgen is measuring (overlay, not transaction processing).

[marta-lokhova]

My vote is for this. It's easier to think in terms of bytes than ops, especially given that we're primarily stressing the overlay with these tests. I'm curious what @marta-lokhova thinks though.

Yeah I agree! This way we also don't mess with apply time measurement by adding extra operations.

[drebelsky]

Regarding using bytes: doing something that is conceptually like adding a padding pseudo op is useful (stellar/stellar-xdr#282), but directly modifying Stellar-transaction.x is suboptimal for downstream users
(stellar/stellar-xdr#282 (comment)). We have a few options (in roughly descending order of my view on the option):

• We could do some build script changes so that we use an alternative xdr file when BUILD_TESTS is enabled. This has the benefit that we can do this change and others without affecting downstream users. Potentially, we can also do changes in stellar-core related to xdr without needing to touch the xdr repo (if this is something we want to do).

  • One consideration is that we'll need to make sure that the rust subsystem uses the same xdr.
  • It's also worth considering how we would keep the xdr-testing modifications mostly representative of the real xdr (e.g., just assume we'll do it correctly manually, have some testing that checks we only have the expected patch, use some sort of patch language/testing, etc…).
  • This could potentially also let us separate core-internal xdr from the stellar-xdr repo.

• We could specify transaction sizes in operations. To avoid the overhead of extra payment TX operations, we could consider using SetOptions (like PRETEND mode currently does).

• We could use SetOptions as a pseudo padding operation. This makes the byte conversion a little more annoying (see, e.g.,

  stellar-core/src/simulation/TxGenerator.cpp

  Line 223 in 70b280f

  if (pretend)

  and

  stellar-core/src/simulation/TxGenerator.cpp

  Line 1257 in 70b280f

  TxGenerator::pretendTransaction(uint32_t numAccounts, uint32_t offset,

  and compare to https://github.com/drebelsky/stellar-core/blob/e57db5763967a8d73b4df591ca44a4db0ae51d0d/src/simulation/TxGenerator.cpp#L1285).

• We could add another type to Stellar-internal.x and add some BUILD_TESTS dependent (de)serialization paths. This seems rather error prone to me.

• Try to get a change to xdrpp like Add padding transaction op for testing stellar-xdr#282 (comment) mentions

  Is there a way to be able to do this without introducing a new operation? Maybe a development feature of xdrpp to be able to specify during codegen a union type that gets additional padding arm for testing? Presumably this would be a useful thin to be able to do to any type.

  xdrpp isn't an SDF repo, so this has some logistical issues (e.g., Is it useful for upstream or just us? If it's only useful for us, is the value of this change worth the maintenance burden of having our own fork?).

• We could have core-testing specific tags in the xdr repo, but this also seems pretty error-prone/an unnecessary maintenance burden to me.

• probably some others that I didn't immediately think of

[drebelsky]

Our current plan is to use the soroban extension in place of a padding op.

[SirTyson]

xdrpp isn't an SDF repo, so this has some logistical issues (e.g., Is it useful for upstream or just us? If it's only useful for us, is the value of this change worth the maintenance burden of having our own fork?).

FYI is this our repo, not sure if it changes the decision though