Enable switching off CDDS extract

[mo-nikosbaltas]

Closes #282.

PR creation checklist for the developer

• Has <issue_number> above ☝️ been replaced with the issue number?
• Has main been selected as the base branch?
• Does the feature branch name follow the format <issue_number>_<short_description_of_feature>?
• Does the text of the PR title exactly match with the text (not including the issue number) of the issue title?
• Have appropriate reviewers been added to the PR (once it is ready for review)?
• Has the PR been assigned to the developer(s)?
• Have the same labels as on the issue (except for the good first issue label) been added to the PR?
• Has the Climate Model Evaluation Workflow (CMEW) project been added to the PR?
• Has the appropriate milestone been added to the PR?

Definition of Done for the developer

• Does the change in this PR address the above issue / have all acceptance criteria been met?
• Does the change in this PR follow the requirements in the wiki: Developer Guide (including copyrights)?
• Have new tests related to the change been added?
• Do all the GitHub workflow checks pass?
• Do all the tests run locally and pass? (Note: the tests are not run by the GitHub workflow, see wiki: Run the tests locally)
• Has the API documentation (e.g. docstrings in Python modules) related to the change been updated appropriately?
• Has the user documentation (i.e. everything in the doc directory) related to the change been updated appropriately, including the Quick Start section?
• Do the HTML pages render correctly? (See wiki: Build the documentation locally)

PR creation checklist for the reviewer

• Has <issue_number> above ☝️ been replaced with the issue number?
• Has main been selected as the base branch?
• Does the feature branch name follow the format <issue_number>_<short_description_of_feature>?
• Does the text of the PR title exactly match with the text (not including the issue number) of the issue title?
• Have appropriate reviewers been added to the PR (once it is ready for review)?
• Has the PR been assigned to the developer(s)?
• Have the same labels as on the issue (except for the good first issue label) been added to the PR?
• Has the Climate Model Evaluation Workflow (CMEW) project been added to the PR?
• Has the appropriate milestone been added to the PR?

Definition of Done for the reviewer

• Does the change in this PR address the above issue / have all acceptance criteria been met?
• Does the change in this PR follow the requirements in the wiki: Developer Guide (including copyrights)?
• Have new tests related to the change been added?
• Do all the GitHub workflow checks pass?
• Do all the tests run locally and pass? (Note: the tests are not run by the GitHub workflow, see wiki: Run the tests locally)
• Has the API documentation (e.g. docstrings in Python modules) related to the change been updated appropriately?
• Has the user documentation (i.e. everything in the doc directory) related to the change been updated appropriately, including the Quick Start section?
• Do the HTML pages render correctly? (See wiki: Build the documentation locally)

[NParsonsMO]

I'm still working through the actual code changes, but have commented on some copyright stuff (and didn't want to lose this progress over lunch).

[NParsonsMO]

Does the text of the PR title exactly match with the text (not including the issue number) of the issue title?

Not exactly but very close (Nikos changed that)

Does the change in this PR address the above issue / have all acceptance criteria been met?

I don't feel qualified to say (Nikos: If you run the unit tests, they should complete successfully. Also, try all three cases (ACs) on your terminal with -O metoffice -O unittest, and should run for the different settings)

Does the change in this PR follow the requirements in the wiki: Developer Guide (including copyrights)?

I've ticked this as I think it does

Has the API documentation (e.g. docstrings in Python modules) related to the change been updated appropriately?

I've ticked this as I agree it didn't need to change

Has the user documentation (i.e. everything in the doc directory) related to the change been updated appropriately, including the Quick Start section?
Do the HTML pages render correctly? (See wiki: Build the documentation locally)

Nothing in doc has changed so I haven't ticked these

[NParsonsMO]

AC1: I agree that when run with EXTRACT=true, as is default, the workflow runs as expected.

AC2: I cannot make this work. I have tried copying previous data (using HOUSEKEEPING=false) to my datadir, as I'm not sure what "the same filesystem" means but this seemed like a likely place for a scientist to have data, but I just get failures in the CDDS workflow at validate_extract_apm. I think there needs to be clearer guidance on how to get this working.

AC3: I agree that the CMEW workflow fails when EXTRACT=false and EXTRACT_DATA_PATH="". It would be nice if it complained earlier, which might be possible using the GUI? But it does seem to do as required if this is not possible.

I have also raised queries about:
a) the naming of variables, and suggest referring to "raw" data
b) the unit tests, which I think are now set to run themselves? But all the other unit tests in the repo have to be run explicitly, so I'm not sure this is required? It's entirely possible I have misunderstood this.

[mo-nikosbaltas]

All requested changes were made, including the renaming of EXTRACT_ to RAW_ in different files.

[mo-nikosbaltas]

I appreciate that the instructions on #282 on how to run for AC2 were confusing and somehow incorrect as I was adding stuff as I was going along. I have updated these instructions and I would like you to review by making a run as instructed there. Just follow from the heading UPDATED INSTRUCTIONS for AC2. This will be part of your review.

[NParsonsMO]

I am approving on the grounds that the raw data variables are now named as I requested, and that I can get the scenario described in AC2 working, using the step-by-step instructions here: #282 (comment)

I have left a question about the HOUSEKEEPING option, but I believe that @ehogan was involved in discussions about this anyway, so I don't feel that's my call.

[mo-nikosbaltas]

Changed HOUSEKEEPING=true 25547ab