Implement a way to clean JcrJobRepository data older than X

[jazeren1]

JcrJobRepository currently does not delete any job statistics from JCR. We should delete any JcrJobRepository data older than X (may be keep only the last successful data for every path)

[sagarsane]

@jbornemann

[jbornemann]

@sagarsane @jdigger seeing this as an issue on our DWH servers. Should probably optimize our status persistence structure, and status computation too. Current status computation I believe is O(n^2), with O(n) data access; since jobExecution, jobInstance, etc trees are 1-level.

With 5000 nodes for each status symbol (execution, instance, etc), it's been reported that status computation takes on the order of days :)

[sagarsane]

Yeah.

Not sure how complex changing the persistence structure would be but at the very least, it should be possible to setup something that deletes older "execution data" than some date .. or like past 2 months or something, or give an ad-hoc option as well.

The trick is to find out :

For a JobInstance, what are the JobExecutions, what are its StepExecutions and what are the Job and Step Execution contexts ... so that the cleanup can be complete.

The Spring batch JobExplorer API I believe has the necessary tools to mine this information.

[jbornemann]

Yeah. Hopefully we could model our store so that that information comes naturally. Consolidating information based on date based nodes may be nice; Maybe something like root -> year -> month -> day -> (jobExecution -> (stepExecution -> stepExecutionContext)+ && jobInstance)*

Could probably help at the code level to implement some memoization

[sagarsane]

Fixed by #107