Prepare for v3

[yulric]

The purpose of this PR is to prepare the dev branch for merging the changes in from the v3.0.0 branch. To that end it does the following:

1. Adds the new columns from the v3.0.0 branch into the variables and variable details sheet in the same order
   • 351bfac45d03f09aef9bc38fa89617ec0cc18980
   • 7af461471c69be7bab93c6a62785dd277f59c98c
   • 7c84d6f61a758af93adc71459f01cc90932ce3e3
   • 27cc85f1b150374defaa061b214d759920b41b46
   • d7fc97d7816d209e405c9e518ca2b7930c32f964
   • ca89d96eed2f0359b7ac0311680bd0a154b54574
   • 666e7e6b05d71dc46a1fe4ea520488a24771b7b6
   • 5e028b6753b0a8bf36d347c20a49e49cb5cd54ce
   • 1a015724890e02a02ed9678da0a4cb2cc1de51d4
2. Adds a new set of functions to check and fix formatting issues in a worksheet.
   • For the reasoning behind the functions
   • For documentation around this feature
   • The remaining commits have the implementation

[DougManuel]

A few comments:

1. Potentially missing columns - The YAML schemas in inst/metadata/schemas/core/variables.yaml include notes in the expected order, and variable_details.yaml includes templateVariable as an optional field.

Should these be added?

2. If you remember, I drafted a similar validation for my work-in-progress. That is in feature/csv-standardisation-updates. Here is a comparison of features, in case you wanted to look that the validations that I included. The pattern, enum and cross-fields were errors that I was dealing with when reviewing and creating code.

Feature	csv-standardisation-updates	This PR
Column order validation	✅	✅
Row sorting validation	❌	✅
Line ending validation (LF/CRLF)	✅	✅
Excessive quoting detection	❌	✅
Trailing empty columns	❌	✅
Pattern validation (regex)	✅	❌
Enum validation	✅	❌
Cross-field validation	✅	❌
Auto-fix capability	✅	✅
GitHub Action	❌	✅

3. I validated using the rules in The YAML schemas in inst/metadata/schemas/core/variables.yaml for column names, ordering, etc. dynamically, rather than hardcoding. The "source of truth" is an interesting discussion.

I also drafted a more extensive report inspired by pkgdown and devtools. I like your error reporting very much, but take a look that that branch if you are interested.

Minor Items

• There's a capture.output(print(row_being_checked), file = "log.txt", append = TRUE) in recode-with-table.R:932 that looks like debug code - should that be removed?
• readr is used in fix_worksheet() but isn't in DESCRIPTION. I think the validation should be a pernament feature of cchsflow, so adding that to description is a consideration.

[yulric]

@DougManuel Thanks for the review! Couple replies below,

1. Feature disparity between your tool and this one: It looks like your tool has validation features which this tool is missing. The focus of this tool is to check and fix formatting issues. I definitely want to support validation though and have actually started work on that in this PR Add parse_variables_sheet function with validation recodeflow#85.
2. Metadata YAML: There's a lot there and I was thinking for this PR that we would only include the items we need. Right now, it looks like we only need the expected_column_order and the id_column_name fields. I will delete the other items and add them in at a later time as needed. What do you think?

[DougManuel]

1. Feature disparity between your tool and this one: It looks like your tool has validation features which this tool is missing. The focus of this tool is to check and fix formatting issues. I definitely want to support validation though and have actually started work on that in this PR Add parse_variables_sheet function with validation recodeflow#85.

For sure, keep this PR focused. I've actually been creating more checks as I encounter issues, but that is for the future.

[DougManuel]

2. Metadata YAML: There's a lot there and I was thinking for this PR that we would only include the items we need. Right now, it looks like we only need the expected_column_order and the id_column_name fields. I will delete the other items and add them in at a later time as needed. What do you think?

Agree.

Feel free to close the PR.