add ability to optionally sample a percentage of images to a lower environment's queue bucket #4542
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
twrichards
force-pushed
the
percentage-sampling-to-lower-environment
branch
from
2deb8ec to
b01b32b
Compare
…vironment's queue bucket, after they've successfully uploaded via S3 copy operation (quick/cheap) - triggered by presence of new `LOWER_ENVIRONMENT_QUEUE_BUCKET_TO_SAMPLE_INTO` environment variable. Sampling is 1% by default, but can be configured via `s3.sampling.percentage`
twrichards
force-pushed
the
percentage-sampling-to-lower-environment
branch
from
b01b32b to
20736e5
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
… after they've successfully uploaded, then using S3 copy operation (quick/cheap) from the image bucket to the lower environment's queue bucket- triggered by presence of new
LOWER_ENVIRONMENT_QUEUE_BUCKET_TO_SAMPLE_INTOenvironment variable (populated by https://github.com/guardian/editorial-tools-platform/pull/988) .Sampling is 1% by default, but can be configured via
s3.sampling.percentageMeans we can effectively revert https://github.com/guardian/editorial-ftp/pull/94 (see https://github.com/guardian/editorial-ftp/pull/175)
What does this change?
How should a reviewer test this change?
How can success be measured?
Who should look at this?
Tested? Documented?