ROW_DELIMITER in COPY was lost.

[cramja]

We had noticed that older versions of the code (uwquickstep/quickstep-retire) had a ROW_DELIMITER specifiable as optional params for the COPY command. This helps load files that have special row delimiters, other than the default \n character.

For example, a file being loaded to the table with schema (id int, name varchar(20)):

1|marc|
2|mike|
3|matt|

has the row delimiter |\n.

The current repo does not have this feature. It should be re-added.

[cramja]

Looking into the issue, it seems like there might be an opportunity to reuse ParseKeyValue pairs here.

[craig-chasseur]

The ROW_DELIMITER, while potentially useful, wound up having poor
performance. A single-char delimiter, as in Postgres, can be scanned and
handled much more efficiently than any general string.

On Mon, Apr 18, 2016 at 10:51 AM Marc S notifications@github.com wrote:

Looking into the issue, it seems like there might be an opportunity to
reuse ParseKeyValue pairs here.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#176 (comment)

[pateljm]

@craig-chasseur makes a good point, but we recently found that the load performance was far more dependent on a mutex that was acquired on each tuple being inserted. So, once that is fixed (@jianqiao is on it!) we should be in a good shape.

After that PR, I think the extra character overhead won't be a big issue, and I'm in favor of the suggestion that @cramja is making to make the system more robust to input file formats. It is not uncommon to see that extra delimiter in practice (the code to generate the text output is simpler, and my guess is that is why text files often have that extra delimiter).