Skip to content

Clean characters in columns #53

@sbehrends

Description

@sbehrends

While implementing csv-streamify I found out that there was a strange character (or wrongly encoded file) in the CSV that returned on an dirty column name.

example.csv

weird_char,ok_col,some_emoj🤙
1,1,1

The object returning included the exact same string as in the keys.

{
  'weird_char': 1,
  ok_col: 1,
  'some_emoj🤙': 1
}

My request is to add the ability to map or clean this column names to get a cleaner object, like

{
  weird_char: 1,
  ok_col: 1,
  some_emoj: 1
}

One solution would be to add a regex replace in https://github.com/klaemo/csv-stream/blob/master/csv-streamify.js#L56, example

if (state.lineNo === 0) {
  state._columns = state._line.map(col => col.replace(/[^a-zA-Z0-9_]/g,''))
  state.lineNo += 1
  reset()
  return
}

A nicer alternative would be to allow to pass a function as value for columns

const csv = require('csv-streamify')
const parser = csv({
  columns: (cols) => cols.map(col => col.replace(/[^a-zA-Z0-9_]/g,'')),
  objectMode: true,
})

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions