Cst jack performance problem tester #153
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…cific criteria
If we can classify problem conditions by specific criteria in stats
and then check to see if those values apply, maybe we can start
diagnosing or at least pointing to hardware problems.
The basic theory being (e.g. for a faulty NVME):
If one disk has read latency > 5ms
And cluster average read latency > 100usec
And ...
And ...
Then maybe we can build a set of test parameters for each one of
those outcomes.
This is not going to be a small undertaking, and unfortunately
we're stuck with bash. However, if the outline test conditions
and theory can be proven useful, this is a step towards a useful
enhancement.
Ideally the logic/tests that come out of this would make it into
the product somehow.
Wekachecker assumes everything is a shell script. Fortunately it's based on a glob so we can skip it
Wekachecker tries to be the governing over-arching controller, so having a single script that parses and tests multiple conditions separately doesn't fit with that architecture. Therefore make every test self-contained
vrragosta
requested changes
Mar 3, 2025
Contributor
vrragosta
left a comment
vrragosta
left a comment
There was a problem hiding this comment.
I think these are the major changes, but I will re-review once changes are made.
Please note, I added comments to the first file, but they are applicable globally.
|
|
||
| DESCRIPTION="Examine WEKA stats for small IO writes" | ||
| SCRIPT_TYPE="single" | ||
| INTERNAL_REFERENCE="WEKAPP-XXXXX" |
Contributor
There was a problem hiding this comment.
All of these tests, except for 002_faulty_drive.sh, refer to a "fake" internal reference of WEKAPP-XXXXX. Can this be changed to an actual reference, or if there is none, omit it from the output?
| #there's a chance we will need to break this down into very different structures of testing, so | ||
| # keeping DRIVE vs CLUSTER tests very separate at the moment, despite the fact it looks ripe | ||
| # for re-factoring. | ||
| for DRIVE_TEST in ${!DRIVE_TESTS[@]} ; do |
Contributor
There was a problem hiding this comment.
Instead of duplicating (most) of the code for DRIVE_TEST and CLUSTER_TEST, can this logic be turned into a function and called from within the for loops?
| # Now we need to convert VALUE to standardised units. We can't rely on the existence of units(1) or $(systemd-analyze timestamp), unfortunately | ||
| # This means we're going to convert using sed and awk, for which I apologize | ||
| # right now we're only calculating based on units of time, and we only seem to see micro- and milliseconds. | ||
| VALUE_CALC=$(echo ${VALUE} | sed 's! *µs!/1000000!g;s! *Microseconds!/1000000!g;s! *ms!/1000!g;s! *!!g;s!s$!!g;s!%!!g') |
Contributor
There was a problem hiding this comment.
I don't think this is handing the drive stats correctly -- on my test cluster, the values are reported in units of Operations/Sec. Trace output:
++ weka stats --show-internal --stat ssd/DRIVE_IO_TOO_LONG --param 'disk:*' --sort value --start-time -2m --output value --raw-units
++ tail -n 1
+ HIGHEST_VALUE='0.0166667 Operations/Sec'
++ convert_to_standard_units '0.0166667 Operations/Sec'
++ local 'VALUE=0.0166667 Operations/Sec'
+++ echo 0.0166667 Operations/Sec
+++ sed 's! *µs!/1000000!g;s! *Microseconds!/1000000!g;s! *ms!/1000!g;s! *!!g;s!s$!!g;s!%!!g'
++ VALUE_CALC=0.0166667Operations/Sec
+++ echo
+++ awk '{print 0.0166667Operations/Sec}'
awk: cmd. line:1: (FILENAME=- FNR=1) fatal: division by zero attempted
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Separate directory so should be entirely merge-safe