merge schema for files which have different number of columns

[minnieshi]

https://github.com/crealytics/spark-excel/wiki/Examples:-Load-Multiple-Files

shows me i can read excel files from a partition;

question is can i somehow tell the lib to merge the schema if some files have extra columns?
what if column sequence are different?
would specify my own schema help?

23/03/02 17:23:13 WARN ExcelHeaderChecker: Number of column in Excel header is not equal to number of fields in the schema: Header length: 62, schema size: 0 Excel file: file:///blah=xyz/day=1/Dummy_Data_catalog_v1.xlsm 23/03/02 17:23:13 INFO FilePartitionReader: Reading file path: file:///blah=xyz/day=1/Dummy_Data_catalog_v1.xlsm, range: 0-28698, partition values: [] 23/03/02 17:23:13 WARN ExcelHeaderChecker: Number of column in Excel header is not equal to number of fields in the schema: Header length: 63, schema size: 0 Excel file: file:///blah=xyz/day=2/Dummy_Data_catalog_v1%20copy.xlsm 23/03/02 17:23:13 INFO FilePartitionReader: Reading file path: file:///blah=xyz/day=2/Dummy_Data_catalog_v1%20copy.xlsm

[nightscape]

@quanghgx might have an answer 😃

[minnieshi]

@quanghgx

[minnieshi]

I had to use force schema and provided schema myself; i wish the code could infer schema if: file 1 has column 1-3 file 2 has column 1-5 (ie, 2 extra columns)

…

On Fri, Mar 3, 2023 at 4:50 PM Martin Mauch ***@***.***> wrote: @quanghgx <https://github.com/quanghgx> might have an answer 😃 — Reply to this email directly, view it on GitHub <#718 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACWMMBRZVGC7646D7HFXOVTW2IHN5ANCNFSM6AAAAAAVNTFMSM> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- Kind regards Min

[gaya3dk2490]

@nightscape @quanghgx Any update on this? with infer schema, reader simply does not merge schema on read. Used different spark versions