SNOW-3077601: Filter on empty dataframe results in schemaless dataframe

[DPeled]

Please answer these questions before submitting your issue. Thanks!

1. What version of Python are you using?

   Python 3.11.13

2. What are the Snowpark Python and pandas versions in the environment?

   pandas==2.1.4
   pandas-gbq==0.31.0
   snowflake-snowpark-python==1.42.0
   

3. What did you do?

   When I created an empty dataframe with 2 columns. I tried to filter based on one of the column, but instead of returning a duplication of that empty dataframe, it returns a dataframe with no columns at all.

   empty_schema = StructType([
            StructField("entry", StringType(), True),
            StructField("file_path", StringType(), True),
        ])
   mock_encounters_df = session.create_dataframe([], schema=empty_schema)
   
   mock_encounters_df.show()
   print("--------------- filter -----------------")
   mock_encounters_df.filter(col("entry").is_null()).show()
   
   # -------------------------
   # |"ENTRY"  |"FILE_PATH"  |
   # -------------------------
   # |         |             |
   # -------------------------
   # --------------- filter -----------------
   # -
   # ||
   # -
   # ||
   # -

4. What did you expect to see?

   The expectation is to see the same dataframe I used before the filter so the output will be:

   -------------------------
   |"ENTRY"  |"FILE_PATH"  |
   -------------------------
   |         |             |
   -------------------------
   --------------- filter -----------------
   -------------------------
   |"ENTRY"  |"FILE_PATH"  |
   -------------------------
   |         |             |
   -------------------------
   

[sfc-gh-sghosh]

Hi @DPeled ,

Thanks for raising the concern.
We are able to reproduce the issue with local session. we will work on eliminating it.

[JoshElkind]

Hi @sfc-gh-sghosh @sfc-gh-jrose @sfc-gh-yixie

I've put together a fix for this issue.

You can check out the PR here: #4084

Root cause: When filtering an empty DataFrame in local testing mode, pandas df[boolean_mask] can return 0 columns due to an index-alignment edge case. The fix preserves the schema by using iloc[0:0].copy() when row filtering incorrectly produces a schemaless result.