slowness issue when sending data into Python-based operators

[mengw15]

summary

Seeing severe latency when data flows from an upstream operator into a Python-based operator. The compute inside Python is fast; the delays happen before and after it, suggesting a transmission/bridge issue.

Example workflow

Workflow A: CSV Scan → Sort (Python)
~3 minutes from workflow start until Sort begins processing
Sort compute itself < 1s
Then ~2 minutes before the workflow reports completion

Workflow B: CSV Scan → Aggregation (Java native)
End-to-end ≈ 20s

Hypothesis

Bottleneck in JVM ↔ Python handoff

[chenlica]

@Yicong-Huang Did you see similar issues in other Apache projects?

[Yicong-Huang]

the serialization is typically even slower in other projects. we are using arrow flight, which is supposed to be fast and almost state of art. Can we debug into what's exactly being slow (e.g., using profiler)?

[Yicong-Huang]

you can also enable DEBUG level log on python side, to see RPC level logs, it will give you more information about the underlying UDF layers such as network.