Skip to content

Conversation

@arjunashok
Copy link
Contributor

Changes

  • Utilized spark listeners to publish job/task stats on job completion for reader and writer.
  • Unified implementation of job end stats published for reader and writer
  • Publishes task runtime stats
  • Adds spark jobGroup based UUID to reader, similar to existing implementation in writer. This is used as the jobId to uniquely identify and potentially merge multiple stats published from the same job.

Testing

  • Validated stats logged from unit tests and in-jvm-dtest integration tests for reader/writer
  • Unit tests for Stats listener

@arjunashok
Copy link
Contributor Author

Closing as duplicate of #51

@arjunashok arjunashok closed this Apr 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant