csvdelimiter and project csv delimiter not consistently respected in exports and imports #491

astruebi · 2025-12-17T17:03:18Z

Summary

This PR makes redcapAPI robust against non-comma CSV delimiters (e.g. semicolon) by:

allowing a connection-level CSV delimiter on redcapApiConnection,
passing csvDelimiter consistently to the API for CSV-based exports,
parsing responses using the same delimiter,
and making writeDataForImport() delimiter-aware for imports.

Details

See issue #1.

High-level changes:

redcapApiConnection: add csv_delimiter(), set_csv_delimiter(), csv_delimiter_api().
exportProjectInformation.redcapApiConnection(), exportDags.redcapApiConnection(), exportRecords*, .exportFieldNamesApiCall(): pass csvDelimiter in the body and parse using sep = rcon$csv_delimiter().
writeDataForImport(): add csv_delimiter argument (default ",") and use write.table(..., sep = csv_delimiter, ...).
importDags.redcapApiConnection(): call writeDataForImport(data, csv_delimiter = rcon$csv_delimiter()).

Testing

Manually verified:
- writeDataForImport() produces valid CSV for "," and ";" delimiters.
- Export helpers use csvDelimiter / sep consistently.

spgarbet · 2025-12-17T18:18:40Z

Can you add your personal ctb details to the DESCRIPTION file for proper attribution?

jubilee2 · 2025-12-17T19:08:33Z

R/redcapConnection.R

+  	csv_delimiter_api <- switch(
+  		csv_delimiter,
+  		","  = "comma",
+  		"\t" = "tab",


the tab should be word "tab" but not "\t" based on document v16.0.4

spgarbet · 2025-12-17T19:47:11Z

I'm seeing failures in our tests from this PR.

══ Results ════════════════════════════════════════════════════════════════
Duration: 166.9 s

── Failed tests ───────────────────────────────────────────────────────────
Failure (test-101-userMethods-ArgumentValidation.R:110:5): Return an error if data does not match User data structure
`importUsers(rcon, data = mtcars)` threw an error with unexpected message.
Expected match: "'names[(]data[)]': Must be a subset of"
Actual message: "1 assertions failed:\n * Variable 'names(data)': Has additional elements {'mpg','cyl','disp','hp','drat','wt','qsec','vs','am','gear','carb'}, but must be a subset of\n * {'username','email','firstname','lastname','expiration','data_access_group','data_access_group_id','data_access_group_label','design','alerts','user_rights','data_access_groups','reports','stats_and_charts','manage_survey_participants','calendar','data_import_tool','data_comparison_tool','logging','email_logging','file_repository','data_quality_create','data_quality_execute','data_quality_resolution','api_export','api_import','api_modules','mobile_app','mobile_app_download_data','record_create','record_rename','record_delete','lock_records_all_forms','lock_records','lock_records_customization','mycap_participants','random_setup','random_dashboard','random_perform','forms','forms_export','record_id_form_access','record_id_export_access','data_export'}."
Backtrace:
    ▆
 1. ├─testthat::expect_error(importUsers(rcon, data = mtcars), "'names[(]data[)]': Must be a subset of") at test-101-userMethods-ArgumentValidation.R:110:5
 2. │ └─testthat:::quasi_capture(...)
 3. │   ├─testthat (local) .capture(...)
 4. │   │ └─base::withCallingHandlers(...)
 5. │   └─rlang::eval_bare(quo_get_expr(.quo), quo_get_env(.quo))
 6. ├─redcapAPI::importUsers(rcon, data = mtcars)
 7. └─redcapAPI:::importUsers.redcapApiConnection(rcon, data = mtcars) at redcapAPI/R/importUsers.R:6:3
 8.   └─checkmate::reportAssertions(coll) at redcapAPI/R/importUsers.R:63:3

Failure (test-102-userRoleMethods-ArgumentValidation.R:81:5): Return an error if data is not a data frame
`importUserRoles(rcon, data = mtcars)` threw an error with unexpected message.
Expected match: "'names[(]data[)]'[:] Must be a subset of"
Actual message: "1 assertions failed:\n * Variable 'names(data)': Has additional elements {'mpg','cyl','disp','hp','drat','wt','qsec','vs','am','gear','carb'}, but must be a subset of\n * {'unique_role_name','role_label','design','user_rights','data_access_groups','reports','stats_and_charts','manage_survey_participants','calendar','data_import_tool','data_comparison_tool','logging','email_logging','file_repository','data_quality_create','data_quality_execute','api_export','api_import','mobile_app','mobile_app_download_data','record_create','record_rename','record_delete','lock_records_customization','lock_records','lock_records_all_forms','mycap_participants','forms','forms_export','random_setup','random_dashboard','random_perform'}."
Backtrace:
    ▆
 1. ├─testthat::expect_error(...) at test-102-userRoleMethods-ArgumentValidation.R:81:5
 2. │ └─testthat:::quasi_capture(...)
 3. │   ├─testthat (local) .capture(...)
 4. │   │ └─base::withCallingHandlers(...)
 5. │   └─rlang::eval_bare(quo_get_expr(.quo), quo_get_env(.quo))
 6. ├─redcapAPI::importUserRoles(rcon, data = mtcars)
 7. └─redcapAPI:::importUserRoles.redcapApiConnection(rcon, data = mtcars) at redcapAPI/R/importUserRoles.R:8:3
 8.   └─checkmate::reportAssertions(coll) at redcapAPI/R/importUserRoles.R:46:3

Failure (test-103-userRoleAssignmentMethods-ArgumentValidation.R:55:5): Return an error if data is not a data frame
`importUserRoleAssignments(...)` threw an error with unexpected message.
Expected match: "'names[(]data[)]': Must be a subset of"
Actual message: "1 assertions failed:\n * Variable 'names(data)': Has additional elements {'user','role'}, but must be a subset of {'username','unique_role_name','data_access_group'}."
Backtrace:
    ▆
 1. ├─testthat::expect_error(...) at test-103-userRoleAssignmentMethods-ArgumentValidation.R:55:5
 2. │ └─testthat:::quasi_capture(...)
 3. │   ├─testthat (local) .capture(...)
 4. │   │ └─base::withCallingHandlers(...)
 5. │   └─rlang::eval_bare(quo_get_expr(.quo), quo_get_env(.quo))
 6. ├─redcapAPI::importUserRoleAssignments(...)
 7. └─redcapAPI:::importUserRoleAssignments.redcapApiConnection(...) at redcapAPI/R/importUserRoleAssignments.R:8:3
 8.   └─checkmate::reportAssertions(coll) at redcapAPI/R/importUserRoleAssignments.R:40:3

Failure (test-103-userRoleAssignmentMethods-ArgumentValidation.R:60:5): Return an error if data is not a data frame
`importUserRoleAssignments(...)` threw an error with unexpected message.
Expected match: "'data[$]username': Must be a subset of"
Actual message: "1 assertions failed:\n * Variable 'data$username': Has additional elements {'not a user'}, but must be a subset of {'beckca','garbetsp','obregos'}."
Backtrace:
    ▆
 1. ├─testthat::expect_error(...) at test-103-userRoleAssignmentMethods-ArgumentValidation.R:60:5
 2. │ └─testthat:::quasi_capture(...)
 3. │   ├─testthat (local) .capture(...)
 4. │   │ └─base::withCallingHandlers(...)
 5. │   └─rlang::eval_bare(quo_get_expr(.quo), quo_get_env(.quo))
 6. ├─redcapAPI::importUserRoleAssignments(...)
 7. └─redcapAPI:::importUserRoleAssignments.redcapApiConnection(...) at redcapAPI/R/importUserRoleAssignments.R:8:3
 8.   └─checkmate::reportAssertions(coll) at redcapAPI/R/importUserRoleAssignments.R:52:3

Failure (test-103-userRoleAssignmentMethods-ArgumentValidation.R:65:5): Return an error if data is not a data frame
`importUserRoleAssignments(...)` threw an error with unexpected message.
Expected match: "'data[$]unique_role_name': Must be a subset of"
Actual message: "2 assertions failed:\n * Variable 'data$username': Has additional elements {'not a user'}, but must be a subset of {'beckca','garbetsp','obregos'}.\n * Variable 'data$unique_role_name': Has additional elements {'not a role'}, but must be a subset of {'NA'}."
Backtrace:
    ▆
 1. ├─testthat::expect_error(...) at test-103-userRoleAssignmentMethods-ArgumentValidation.R:65:5
 2. │ └─testthat:::quasi_capture(...)
 3. │   ├─testthat (local) .capture(...)
 4. │   │ └─base::withCallingHandlers(...)
 5. │   └─rlang::eval_bare(quo_get_expr(.quo), quo_get_env(.quo))
 6. ├─redcapAPI::importUserRoleAssignments(...)
 7. └─redcapAPI:::importUserRoleAssignments.redcapApiConnection(...) at redcapAPI/R/importUserRoleAssignments.R:8:3
 8.   └─checkmate::reportAssertions(coll) at redcapAPI/R/importUserRoleAssignments.R:52:3

Failure (test-104-dagMethods-ArgumentValidation.R:55:5): Return an error if data is not a data.frame
`importDags(rcon, data = data.frame(bad_var = "something"))` threw an error with unexpected message.
Expected match: "'names[(]data[)]': Must be a subset of"
Actual message: "1 assertions failed:\n * Variable 'names(data)': Has additional elements {'bad_var'}, but must be a subset of {'data_access_group_name','unique_group_name','data_access_group_id'}."
Backtrace:
    ▆
 1. ├─testthat::expect_error(...) at test-104-dagMethods-ArgumentValidation.R:55:5
 2. │ └─testthat:::quasi_capture(...)
 3. │   ├─testthat (local) .capture(...)
 4. │   │ └─base::withCallingHandlers(...)
 5. │   └─rlang::eval_bare(quo_get_expr(.quo), quo_get_env(.quo))
 6. ├─redcapAPI::importDags(rcon, data = data.frame(bad_var = "something"))
 7. └─redcapAPI:::importDags.redcapApiConnection(rcon, data = data.frame(bad_var = "something")) at redcapAPI/R/importDags.R:8:3
 8.   └─checkmate::reportAssertions(coll) at redcapAPI/R/importDags.R:40:3

Failure (test-104-dagMethods-ArgumentValidation.R:67:5): Return an error if a unique_group_name isn't in the project
`importDags(rcon, data = FailDag)` threw an error with unexpected message.
Expected match: "'data[$]unique_group_name': Must be a subset of"
Actual message: "1 assertions failed:\n * Variable 'data$unique_group_name': Has additional elements {'bad name'}, but must be a subset of {'NA'}."
Backtrace:
    ▆
 1. ├─testthat::expect_error(importDags(rcon, data = FailDag), "'data[$]unique_group_name': Must be a subset of") at test-104-dagMethods-ArgumentValidation.R:67:5
 2. │ └─testthat:::quasi_capture(...)
 3. │   ├─testthat (local) .capture(...)
 4. │   │ └─base::withCallingHandlers(...)
 5. │   └─rlang::eval_bare(quo_get_expr(.quo), quo_get_env(.quo))
 6. ├─redcapAPI::importDags(rcon, data = FailDag)
 7. └─redcapAPI:::importDags.redcapApiConnection(rcon, data = FailDag) at redcapAPI/R/importDags.R:8:3
 8.   └─checkmate::reportAssertions(coll) at redcapAPI/R/importDags.R:47:3

Failure (test-105-dagAssignment-ArgumentValidation.R:64:5): Return an error if data doesn't have the right structure
`importUserDagAssignments(rcon, data = BadData)` threw an error with unexpected message.
Expected match: "'names[(]data[)]': Must be a subset of"
Actual message: "1 assertions failed:\n * Variable 'names(data)': Has additional elements {'bad_var','not_dag'}, but must be a subset of {'username','redcap_data_access_group'}."
Backtrace:
    ▆
 1. ├─testthat::expect_error(...) at test-105-dagAssignment-ArgumentValidation.R:64:5
 2. │ └─testthat:::quasi_capture(...)
 3. │   ├─testthat (local) .capture(...)
 4. │   │ └─base::withCallingHandlers(...)
 5. │   └─rlang::eval_bare(quo_get_expr(.quo), quo_get_env(.quo))
 6. ├─redcapAPI::importUserDagAssignments(rcon, data = BadData)
 7. └─redcapAPI:::importUserDagAssignments.redcapApiConnection(...) at redcapAPI/R/importUserDagAssignments.R:9:3
 8.   └─checkmate::reportAssertions(coll) at redcapAPI/R/importUserDagAssignments.R:39:3

Failure (test-105-dagAssignment-ArgumentValidation.R:71:5): Return an error if data doesn't have the right structure
`importUserDagAssignments(rcon, data = BadData)` threw an error with unexpected message.
Expected match: "'data[$]username': Must be a subset of"
Actual message: "2 assertions failed:\n * Variable 'data$username': Has additional elements {'username'}, but must be a subset of {'beckca','bstat_api_user','garbetsp','obregos'}.\n * Variable 'data$redcap_data_access_group': Has additional elements {'dag_name'}, but must be a subset of {'NA'}."
Backtrace:
    ▆
 1. ├─testthat::expect_error(...) at test-105-dagAssignment-ArgumentValidation.R:71:5
 2. │ └─testthat:::quasi_capture(...)
 3. │   ├─testthat (local) .capture(...)
 4. │   │ └─base::withCallingHandlers(...)
 5. │   └─rlang::eval_bare(quo_get_expr(.quo), quo_get_env(.quo))
 6. ├─redcapAPI::importUserDagAssignments(rcon, data = BadData)
 7. └─redcapAPI:::importUserDagAssignments.redcapApiConnection(...) at redcapAPI/R/importUserDagAssignments.R:9:3
 8.   └─checkmate::reportAssertions(coll) at redcapAPI/R/importUserDagAssignments.R:51:3

Failure (test-105-dagAssignment-ArgumentValidation.R:75:5): Return an error if data doesn't have the right structure
`importUserDagAssignments(rcon, data = BadData)` threw an error with unexpected message.
Expected match: "'data[$]redcap_data_access_group': Must be a subset of"
Actual message: "2 assertions failed:\n * Variable 'data$username': Has additional elements {'username'}, but must be a subset of {'beckca','bstat_api_user','garbetsp','obregos'}.\n * Variable 'data$redcap_data_access_group': Has additional elements {'dag_name'}, but must be a subset of {'NA'}."
Backtrace:
    ▆
 1. ├─testthat::expect_error(...) at test-105-dagAssignment-ArgumentValidation.R:75:5
 2. │ └─testthat:::quasi_capture(...)
 3. │   ├─testthat (local) .capture(...)
 4. │   │ └─base::withCallingHandlers(...)
 5. │   └─rlang::eval_bare(quo_get_expr(.quo), quo_get_env(.quo))
 6. ├─redcapAPI::importUserDagAssignments(rcon, data = BadData)
 7. └─redcapAPI:::importUserDagAssignments.redcapApiConnection(...) at redcapAPI/R/importUserDagAssignments.R:9:3
 8.   └─checkmate::reportAssertions(coll) at redcapAPI/R/importUserDagAssignments.R:51:3

[ FAIL 10 | WARN 0 | SKIP 0 | PASS 590 ]
══ Terminated early ═══════════════════════════════════════════════════════

spgarbet · 2025-12-17T19:57:13Z

These errors seem unrelated. Has checkmate pushed an update?

Update: I tested main. I have the github version of checkmate installed. I reverted to CRAN. These errors are occurring with an unpublished version of checkmate. When they push to CRAN we will need to update our test suite.

spgarbet · 2025-12-17T20:19:05Z

══ Results ════════════════════════════════════════════════════════════════
Duration: 627.3 s

── Skipped tests (14) ─────────────────────────────────────────────────────
• No Data Quality Project provided. Tests Skipped (1):
  test-904-exportDataQuality.R:4:3
• No logs for test window in past (1):
  test-800-exportLogging-Functionality.R:16:5
• No value provided for EXPORT_REPORTS_ID. Tests Skipped (3):
  test-240-exportTypedReports-Functionality.R:6:5,
  test-240-exportTypedReports-Functionality.R:52:5,
  test-240-exportTypedReports-Functionality.R:70:5
• Tests to be developed for 3.0.0 release (9):
  test-151-exportRecords-ArgumentValidation.R:4:5,
  test-151-exportRecords-Functionality.R:4:5,
  test-152-exportRecordsWithDags.R:4:5,
  test-153-exportRecordsWithEvents.R:4:5,
  test-155-exportRecordsWithRepeatingInstruments.R:4:5,
  test-180-exportRecordsOffline-ArgumentValidation.R:4:5,
  test-180-exportRecordsOffline-Functionality.R:4:5,
  test-190-exportReports-ArgumentValidation.R:4:5,
  test-190-exportReports-Functionality.R:4:5

[ FAIL 0 | WARN 0 | SKIP 14 | PASS 2051 ]

Tests on behavior we depend on are all passing.

spgarbet · 2025-12-17T20:19:54Z

Okay, summary:

Need to update DESCRIPTION for contribution attribution.
Fix the tab issue mentioned by Jubilee

and this looks good to me.

astruebi · 2025-12-18T10:25:33Z

It's easier than I thought. There's no need to distinguish between csvDelimiter and csvDelimiterApi. REDCap can also handle getting the real delimiter character in body of makeApiCall(...).
I manually tested all delimiters with REDCap 15.3.2 — including the whitespace character and all the functions I changed.

spgarbet · 2025-12-18T14:58:10Z

No regression

[ FAIL 0 | WARN 0 | SKIP 14 | PASS 2051 ]

and

── R CMD check results ───────────────────────── redcapAPI 2.11.5.9000 ────
Duration: 1m 38s

❯ checking for future file timestamps ... NOTE
  unable to verify current time

0 errors ✔ | 0 warnings ✔ | 1 note ✖

R CMD check succeeded

astruebi added 3 commits December 17, 2025 15:02

Add CSV delimiter get/set to redcapConnection

c99abd0

Fix CSV delimiter in (some) export functions

4e5db14

Fix CSV delimiter in (some) import functions

496746b

jubilee2 reviewed Dec 17, 2025

View reviewed changes

spgarbet mentioned this pull request Dec 17, 2025

github edge version of checkmate breaks current test suite. #492

Open

astruebi added 4 commits December 18, 2025 11:06

Remove distinction between csvDelimiter and csvDelimiterApi

34ba473

Add whitespace as a valid CSV delimiter

e39cf5c

Update list of contributors in DESCRIPTION

e0b1bd5

Update .rd files

e3fc092

spgarbet self-requested a review December 18, 2025 14:57

spgarbet approved these changes Dec 18, 2025

View reviewed changes

spgarbet merged commit eb76b6d into vubiostat:main Dec 18, 2025
6 of 7 checks passed

spgarbet added a commit that referenced this pull request Dec 18, 2025

Bumped NEWS for PR #491

8bc60dc

This was referenced Dec 19, 2025

Expand MRP Check command line arguments by timestamp for medication date proxy medizininformatik-initiative/INTERPOLAR#1087

Open

db2fronted: exportRecords -> exportRecordsTyped medizininformatik-initiative/INTERPOLAR#1025

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

csvdelimiter and project csv delimiter not consistently respected in exports and imports #491

csvdelimiter and project csv delimiter not consistently respected in exports and imports #491

Uh oh!

astruebi commented Dec 17, 2025

Uh oh!

spgarbet commented Dec 17, 2025

Uh oh!

jubilee2 Dec 17, 2025

Uh oh!

spgarbet commented Dec 17, 2025

Uh oh!

spgarbet commented Dec 17, 2025 •

edited

Loading

Uh oh!

spgarbet commented Dec 17, 2025

Uh oh!

spgarbet commented Dec 17, 2025

Uh oh!

astruebi commented Dec 18, 2025

Uh oh!

spgarbet commented Dec 18, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

csvdelimiter and project csv delimiter not consistently respected in exports and imports #491

csvdelimiter and project csv delimiter not consistently respected in exports and imports #491

Uh oh!

Conversation

astruebi commented Dec 17, 2025

Summary

Details

Testing

Uh oh!

spgarbet commented Dec 17, 2025

Uh oh!

jubilee2 Dec 17, 2025

Choose a reason for hiding this comment

Uh oh!

spgarbet commented Dec 17, 2025

Uh oh!

spgarbet commented Dec 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

spgarbet commented Dec 17, 2025

Uh oh!

spgarbet commented Dec 17, 2025

Uh oh!

astruebi commented Dec 18, 2025

Uh oh!

spgarbet commented Dec 18, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

spgarbet commented Dec 17, 2025 •

edited

Loading