feat: Add flattening_separator as an additional flattening config

[iamlorax]

Description

This PR allows users to set the flattening_separator config along side the other flattening options. It is used override the default separator of "__".

Related Issues

Closes #3401

Summary by Sourcery

Add support for configuring a custom flattening separator and validate it in flattening utilities.

New Features:

• Allow plugins to specify a custom flattening_separator in flattening configuration to override the default separator.

Tests:

• Add tests to verify get_flattening_options accepts flattening_separator and that flatten_schema and flatten_record honor a custom separator value.

[sourcery-ai]

Reviewer's Guide

Adds support for a configurable flattening_separator in flattening options and extends flattening helpers and tests to honor a custom separator instead of the hard-coded '__'.

Class diagram for updated flattening configuration support

classDiagram
    class PluginFlatteningConfig {
        <<TypedDict>>
        bool flattening_enabled
        int flattening_max_depth
        int flattening_max_key_length
        str flattening_separator
    }

    class FlatteningOptions {
        bool enabled
        int max_depth
        int max_key_length
        str separator
        +from_dict(cls, data: PluginFlatteningConfig) FlatteningOptions
    }

    PluginFlatteningConfig --> FlatteningOptions : used_by_from_dict

Loading

File-Level Changes

Change	Details	Files
Allow flattening options to accept and propagate a custom separator value.	Extend PluginFlatteningConfig typed dict to include a flattening_separator string field. Update FlatteningOptions.from_dict to read flattening_separator from config and pass it as separator when constructing FlatteningOptions instances.	singer_sdk/helpers/_flattening.py
Verify flattening functions respect a custom separator in options and explicit parameters.	Add a unit test ensuring get_flattening_options reads flattening_separator and maps it to options.separator. Add tests for flatten_schema using a '.' separator to verify flattened property keys are joined with the custom separator and arrays are handled consistently. Add tests for flatten_record using a '.' separator to verify flattened records and serialization of nested objects are correct.	tests/core/test_flattening.py

Assessment against linked issues

Issue	Objective	Addressed	Explanation
#3401	Add a new flattening configuration option to allow users to set a custom flattening separator instead of the hard-coded "__".	✅
#3401	Ensure the flattening logic (schema and record flattening) respects the configured/custom separator value.	✅

Possibly linked issues

• feat: Allow flattening separator to be set via new flattening config #3401: PR implements the issue by adding flattening_separator config and wiring it into flattening options and behavior.

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sourcery-ai]

Hey there - I've reviewed your changes - here's some feedback:

• Consider marking flattening_separator as optional/NotRequired in PluginFlatteningConfig so the type signature matches existing usage where this key is not always present.
• The new custom-separator tests duplicate some of the existing schema/record structures; extracting a shared fixture or helper for these nested structures would reduce repetition and make future changes easier.

Prompt for AI Agents

Please address the comments from this code review:

## Overall Comments
- Consider marking `flattening_separator` as optional/NotRequired in `PluginFlatteningConfig` so the type signature matches existing usage where this key is not always present.
- The new custom-separator tests duplicate some of the existing schema/record structures; extracting a shared fixture or helper for these nested structures would reduce repetition and make future changes easier.

## Individual Comments

### Comment 1
<location> `tests/core/test_flattening.py:349-358` </location>
<code_context>
+def test_flatten_schema_with_custom_separator():
</code_context>

<issue_to_address>
**suggestion (testing):** Add an end-to-end test that wires `flattening_separator` from config through to `flatten_schema`/`flatten_record`.

Current tests show that `flatten_schema`/`flatten_record` respect a custom `separator` and that config is parsed into `FlatteningOptions`, but they don’t cover the full flow from plugin config through `get_flattening_options` to the flattening helpers. Please add an integration-style test that calls `get_flattening_options` with a custom `flattening_separator`, then uses the returned `FlatteningOptions` with `flatten_schema` and/or `flatten_record`, asserting that the flattened keys use the configured separator.

Suggested implementation:

```python
def test_flatten_schema_with_custom_separator():
    """End-to-end test that flattening_separator in config flows through to flattening helpers."""
    # Build a config that specifies a custom flattening separator
    config = {
        "flattening_enabled": True,
        "flattening_max_depth": 10,
        "flattening_separator": "__",
    }

    # Get options from config
    options = get_flattening_options(config)

    # Schema with nested object to verify key flattening
    schema = {
        "type": "object",
        "properties": {
            "key_1": {"type": "string"},
            "key_2": {
                "type": "object",
                "properties": {
                    "key_3": {"type": "string"},
                },
            },
        },
    }

    # Record matching the schema
    record = {"key_1": "value1", "key_2": {"key_3": "value3"}}

    # When flattening using the options derived from config
    flattened_schema = flatten_schema(schema, options=options)
    flattened_record = flatten_record(record, schema=schema, options=options)

    # Then the configured separator is used in the flattened keys
    assert "key_2__key_3" in flattened_schema["properties"]
    assert "key_2__key_3" in flattened_record
    assert flattened_record["key_2__key_3"] == "value3"

```

This change assumes the following existing interfaces, which are consistent with the rest of the tests in this module:
1. `get_flattening_options(config: dict) -> FlatteningOptions`
2. `flatten_schema(schema: dict, options: FlatteningOptions) -> dict`
3. `flatten_record(record: dict, schema: dict, options: FlatteningOptions) -> dict`

If your actual function signatures differ (for example, if `options` is a positional argument or named differently), adjust the calls in the new test accordingly. Also ensure that `get_flattening_options`, `flatten_schema`, and `flatten_record` are imported at the top of `tests/core/test_flattening.py` if they are not already.
</issue_to_address>

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.14%. Comparing base (a6fdc6a) to head (94c077a).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #3415   +/-   ##
=======================================
  Coverage   94.13%   94.14%           
=======================================
  Files          69       69           
  Lines        5782     5785    +3     
  Branches      716      717    +1     
=======================================
+ Hits         5443     5446    +3     
  Misses        236      236           
  Partials      103      103           

Flag	Coverage Δ
core	81.98% <100.00%> (+<0.01%)	⬆️
end-to-end	76.35% <33.33%> (-0.03%)	⬇️
optional-components	43.47% <33.33%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[codspeed-hq]

Merging this PR will not alter performance

Summary

✅ 8 untouched benchmarks

---

_{Comparing iamlorax:feature/add-flattening-separator-config (94c077a) with main (a6fdc6a)}