[ENH] Add TextFeatures transformer for text feature extraction

[ankitlade12]

• Add TextFeatures class to extract features from text columns
• Support for features: char_count, word_count, digit_count, uppercase_count, etc.
• Add comprehensive tests with pytest parametrize
• Add user guide documentation

[solegalli]

Hi @ankitlade12

Thanks a lot!

This transformer, function-wise, I'd say it's ready. I made a few suggestions regarding how to optimize the feature creation functions. Let me know if they make sense.

Other than that, we need the various docs file and we'll be good to go :)

Thanks again!

[solegalli]

We need to rebase main so the 2 remaining tests pass.

[solegalli]

Hi @ankitlade12

I am very sorry for the delayed review. I am travelling till end of April, so I am a bit slower than usual.

I think, for the first version of the transformer, let's enforce the user to pass the names of the text variables. They can pass one or more variables in case there are more than one text column.

Other than that, we need to add the tranformer in the docs/index file, in the readme, and in the docs/api, and adjust the tests and the demo to the newer functionality. Then it is good to merge.

Thank you very much for this great addition.

[ankitlade12]

Hey @solegalli,

I tracked down the cause of the CI failures. They are caused by Pandas 2.2/3.0 breaking changes in the CI environment (specifically datetime string formatting and select_dtypes behavior).

Because these are breaking the entire library (280 failures), they need to be fixed in the main branch first. My
TextFeatures
code works perfectly on the current stable releases and passes all 30 isolated tests. I'd be happy to re-run the CI once the library is updated to handle the new Pandas string/datetime standards."

[solegalli]

Hi @ankitlade12

Something went wrong here. The arcsintransformer files, for some reason are in this PR. We need to remove all commits starting from [update ihs user guide] onwards. Then, we need to make sure that the requested changes (those in conversations marked as resolved) are present in the PR.

Could you please take a look? Thanks a lot!

[solegalli]

I made the tests passing optional, because there is a lot to fix on the pandas 3 side. If you want to go ahead and remove unrelated files, we can merge this transformer while we work on the maintenance fixes.

[codecov]

Codecov Report

❌ Patch coverage is 89.61039% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 98.10%. Comparing base (322c8de) to head (848670f).

Files with missing lines	Patch %	Lines
feature_engine/text/text_features.py	89.33%	6 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #880      +/-   ##
==========================================
- Coverage   98.23%   98.10%   -0.14%     
==========================================
  Files         114      116       +2     
  Lines        4886     4963      +77     
  Branches      772      788      +16     
==========================================
+ Hits         4800     4869      +69     
- Misses         55       61       +6     
- Partials       31       33       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[solegalli]

Hey @ankitlade12

This PR has a bunch of unrelated files. The files corresponding to the text transformer are gone. I suggest resetting hard to the last commit where the files corresponding to the transformer are present, and addressing the comments again from there.

[ankitlade12]

Hi @solegalli,

Thanks for the feedback! I've cleaned up the PR:

Removed unrelated files:

• diagnose_tz_local.py, reproduce_tz_error.py (debugging scripts)
• All arcsinh images (already merged in [ENH] Add ArcSinhTransformer for inverse hyperbolic sine transformation #879)
• Reverted changes to log.py, test_sklearn_wrapper.py, test_check_estimator_transformers.py

Restored TextFeatures transformer:

• feature_engine/text/init.py
• feature_engine/text/text_features.py
• tests/test_text/init.py
• tests/test_text/test_text_features.py

The branch now only contains the TextFeatures implementation with 16 passing tests. I rebased on latest main to resolve the merge conflict.

Ready for re-review!

[solegalli]

Hey @ankitlade12

Thank you so much for the quick turnaround. Unfortunately, now all the user guide and api files are lost. We need to put those back in :). Together with the readme, doc/index, etc.