fix: Spark freqtable and missing values

[MCBoarder289]

This PR addresses issue #1429, as well as some of the histogram issues in #1602. When trying to resolve the missing category in the charts when Spark is used as a backend, I discovered a few bugs in the logic computing summary stats, as well as with the aggregations for value counts of not-null records.

I approached this by creating a toy dataset of both strings and numbers to profile in both Spark and Pandas and compare the reports afterwards. The toy dataset includes nulls on both fields, as well as duplicate records.

Initial Pandas Profile Output

• Summary Stats:
  
• Common Values:
  

Initial Spark Profile Output

• Summary Stats:
  

• Common Values:
  

Issues and Root Causes
There are couple of commits in here that address specific root causes of these discrepancies. Here are the summarized issues with their solutions:

• Issue 1: pandas by default will count "NaN" values as Null in summary stats, but Spark SQL does not, so we explicitly address that in one of the commits.

  • Resolution: This was resolved by ensuring the numeric_stats_spark() method explicitly filters out nulls and Nans to match pandas' default behavior

• Issue 2: Missing values were not being properly calculated because NaN in Spark is not null, so they weren't considered missing when they should be

  • Resolution: Adding nan filters to the n_missing computation in the describe_spark_counts()method

• Issue 3: Histogram counts and Common Values counts using the summary["value_counts_without_nan"] Series were not correctly summing counts.

  • Resolution: Adding a sum to the counts, and removing the limit(200) makes everything line up to parity with the Pandas output
  • NOTE: Since we're pre-aggregating data for the value_counts, I don't think the limit(200) is necessary even with Spark. Since we're pulling this down into a Pandas Series anyway, if the data was too big, then that would explicitly fail the process instead of producing misleading reports. If you're running this in Spark on big data anyway, it's a reasonable assumption that you're using a high-memory compute instance anyway.

Fixed Spark Profile Output

• Summary Stats:
  

• Common Values:
  

Concluding Thoughts
While there is still some very slight variation to the computed stats because of how Spark handles nulls/NaNs differently than pandas, I think this new output is acceptably close to the pandas version and any differences are ultimately negligible. Especially when comparing the initial outputs where the differences are misleading without these fixes.

@fabclmnt - I definitely welcome any and all feedback on this approach! I'm happy to discuss further, and hope this is helpful to anyone using the Spark backend. I think this would knock out a few bug tickets overall.

[MCBoarder289]

Closing in favor of #1800 which fixes multiple issues at once