Issue with flatten function when some records contain nested values and others are blank

[allanw]

The flatten function (

target-csv/target_csv.py

Line 28 in a98d463

def flatten(d, parent_key='', sep='__'):

) will return a dictionary with the correctly flattened values if a nested value exists. However, if another record in the same input source contains the same key and is blank, the recursive call to flatten will not return anything, and so that key will not be present in the return value.

In the example above, the dictionaries returned from flatten will have different numbers of keys, which causes a problem when writing the output to a CSV (i.e. the number of columns will differ).

[abij]

I think you need to flatten, since it's going to be persisted into a CSV-file.
I ran into same issue, having more fields. I fixed this by by appending new discovered headers on the end of the header. And replace the header at the end of the stream #27.

Maybe this will help you too.