Add pre_condtime for preconditioning and allow multiple randrw workloads within the same YAML

[lee-j-sanders]

This PR only changes the functionality of workloads.

If workloads are in use, and there is a preconditioning step within the YAML, often users will want to run preconditioning for say 10 minutes, where as each other workload will run for a shorter period of time, this PR allows the user to override time with precond_time and specify precond within the preconditioning step to use the different time attribute.

Additionally, workloads with randrw only let you specify 1 mixed workload within a YAML as the same directory name was used for the subsequent randrw workloads, thus overwriting all the previous results in the archive directory with the later tests. This PR adds the readwrite ratio into the directory name if the mode is randrw.

The directory name for 100% read and 100% write tests are unaffected.

[lee-j-sanders]

Note, at the moment this is based on #324 due to changes in the same area - at the time of writing PR324 is unmerged hence the PR324 changes are appearing in the review.

[lee-j-sanders]

================================================================================================== test session starts ==================================================================================================
platform linux -- Python 3.9.21, pytest-8.3.4, pluggy-1.5.0
rootdir: /cbt.lee
collected 331 items

tests/test_benchmarkfactory.py ... [ 0%]
tests/test_bm_cephtestrados.py .................. [ 6%]
tests/test_bm_fio.py ......................................... [ 18%]
tests/test_bm_getput.py .............................. [ 27%]
tests/test_bm_hsbench.py .............................. [ 36%]
tests/test_bm_kvmrbdfio.py ................................... [ 47%]
tests/test_bm_librbdfio.py ..................................................... [ 63%]
tests/test_bm_nullbench.py ............ [ 67%]
tests/test_bm_radosbench.py ............................... [ 76%]
tests/test_bm_rawfio.py ................................ [ 86%]
tests/test_bm_rbdfio.py .................................... [ 96%]
tests/test_common.py .sss [ 98%]
tests/test_common_output_formatter.py ...... [100%]

=================================================================================================== warnings summary ====================================================================================================
post_processing/formatter/test_run_result.py:21
/cbt.lee/post_processing/formatter/test_run_result.py:21: PytestCollectionWarning: cannot collect test class 'TestRunResult' because it has a init constructor (from: tests/test_common_output_formatter.py)
class TestRunResult:

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
======================================================================================= 328 passed, 3 skipped, 1 warning in 0.46s =======================================================================================

The warning looks to be a problem in master with the common output formatter stuff, not anything I've introduced.

I will look into this with @harriscr before merging.

[perezjosibm]

I don't think the new option for precond_time is needed, I invite the submitter to try the suggested experiment without the change proposed to help convince himself.

[perezjosibm]

If you look at the way workloads work, basically they temporarily overwrite the global FIO options defined in the test plan .yaml (normally located before the workloads section). So within a precondition one can set a time value that only affects the precondition, which is then restored to the global. Then, another workload step would do the same.

In other words, there is not actual need to introduce yet another 'precond_time'!

But you can do the experiment: define two different time values in a precond and a workload step, from the global ones, with the existing code (that is, without this change) to convince yourself.

[perezjosibm]

And this is also mentioned in the documentation https://github.com/ceph/cbt/blob/master/docs/Workloads.md

[harriscr]

The pytest warning doesn't stop the test from being run, so all run successfully. We are getting the warning because pytest assumes that anything with test in the title is actually a test case and tries to treat it as such. See the default glob pattern

Interestingly I don't see the warning when running the tests via vscode, only on the cli, which is something I want to investigate more.

There are 2 ways to remove the warning. One is to add __test__ = False to the TestRunResult class itself, and the other would be to set TestRunResult.__test__ = False in the test file (test_common_output_formatter.py)

The second option gives a mypy attr-defined error, so the first seems to be the best way to go.
I have added the change to the total_iodepth PR

[perezjosibm]

I had a quick look: In the constructor, we load the workloads as a dict, if not empty we backup the global FIO options, which may include iodepth, etc -- any option that is used for mkfiocmd().

        self.workloads = config.get('workloads', {})
        if self.workloads:
            self.backup_global_fio_options()

Notice thought that it does not include 'time', (or more appropriate, 'runtime' per workload step), so to make the required runtime to be overloaded by each workload step, simply add the line (or a more appropriate default time) in the constructor -- and corresponding backup/restore class methods:

 self.time = config.get('time', None)

Otherwise, adding yet another option explicitly ('precond_time') will clutter the code.

[lee-j-sanders]

To avoid confusion decided to abandon this pull request and raise a new one.