Work towards automated cleaning of JcrJobRepository by sagarsane · Pull Request #107 · TWCable/grabbit

sagarsane · 2016-04-05T22:40:59Z

No description provided.

sagarsane · 2016-04-06T04:59:13Z

@jazeren1 @jbornemann @jdigger just WIP. Need to still properly test this and implement unit tests

sagarsane · 2016-04-07T16:02:04Z

Opening this up for some initial "approach" review.

Currently I have a servlet that's triggering the cleaning. I'll be updating this to have a Cron like we discussed. Wanted to get some feedback on the approach I have taken.

After I almost finished the job repp cleanup functionality .. But then I realized that there are way too many private methods in the service I have for cleanup. (this commit : 1e51ff8)

Because of that, I thought it'd be better "host" the functionality on the DAO objects themselves.. Felt like that looked and organized better. ( this commit: 7d855e2)

Does the latter approach look better? In order to make that happen, I had to extend Spring Batch's DAO interfaces to add the additional APIs we need for clean-up.

@jdigger @jazeren1 @jbornemann @masroormohammed can you please take a look?

sagarsane · 2016-04-07T16:07:44Z

Ugh hold on. Tests are failing. Taking a look.

sagarsane · 2016-04-07T16:26:23Z

I'll change the logging to debug, etc. when this is good to go. I figured the logging will help right now during testing.

Also made the change in grabbit version right now for easier installation for testing.

To test,

gw clean build install -x uninstallBundles -x uninstall -x remove

I've had to ignore uninstall. Seems to be having issues with other bundles in AEM :(

Run some content syncs for any number of paths .. or if you already have some data under

/var/grabbit/job/repository/jobExecutions then you probably have some data you can use to test the clean up.

Current testing is to do a POST to http://localhost:4502/grabbit/jobrepository/clean and add a hours parameter that should be an integer.

This number can be any "integer" .. I have tested that it works fine for > 24 hours as well. More-over, I am also logging everything that's going on .. along with the Date its using to compare and everything. So your logs will look something like this :

07.04.2016 16:20:01.267 *INFO* [10.0.2.2 [1460046001202] GET /bin/cleanRepository HTTP/1.1] com.twcable.grabbit.spring.batch.repository.JcrJobExecutionDao JobExecutions: [/var/grabbit/job/repository/jobExecutions/1628554003093086836, /var/grabbit/job/repository/jobExecutions/5944394638596509927], size: 2
07.04.2016 16:20:01.269 *INFO* [10.0.2.2 [1460046001202] GET /bin/cleanRepository HTTP/1.1] com.twcable.grabbit.spring.batch.repository.JcrJobExecutionDao Current time: Thu Apr 07 16:20:01 UTC 2016
07.04.2016 16:20:01.269 *INFO* [10.0.2.2 [1460046001202] GET /bin/cleanRepository HTTP/1.1] com.twcable.grabbit.spring.batch.repository.JcrJobExecutionDao Hours 24 .. OlderThanHours Time: Wed Apr 06 16:20:01 UTC 2016
07.04.2016 16:20:01.271 *INFO* [10.0.2.2 [1460046001202] GET /bin/cleanRepository HTTP/1.1] com.twcable.grabbit.spring.batch.repository.JcrJobExecutionDao JobExecutionsOlder than 24 hours: [] , length: 0
07.04.2016 16:20:01.277 *INFO* [10.0.2.2 [1460046001202] GET /bin/cleanRepository HTTP/1.1] com.twcable.grabbit.spring.batch.repository.services.impl.DefaultCleanJobRepository jobInstancesToRemove: [], size: 0
07.04.2016 16:20:01.278 *INFO* [10.0.2.2 [1460046001202] GET /bin/cleanRepository HTTP/1.1] com.twcable.grabbit.spring.batch.repository.services.impl.DefaultCleanJobRepository stepExecutionsToRemove: [], size: 0
07.04.2016 16:20:01.278 *INFO* [10.0.2.2 [1460046001202] GET /bin/cleanRepository HTTP/1.1] com.twcable.grabbit.spring.batch.repository.services.impl.DefaultCleanJobRepository jobExecutionContextsToRemove: [], size: 0
07.04.2016 16:20:01.278 *INFO* [10.0.2.2 [1460046001202] GET /bin/cleanRepository HTTP/1.1] com.twcable.grabbit.spring.batch.repository.services.impl.DefaultCleanJobRepository stepExecutionContextsToResource: [], size: 0
07.04.2016 16:20:01.278 *INFO* [10.0.2.2 [1460046001202] GET /bin/cleanRepository HTTP/1.1] com.twcable.grabbit.spring.batch.repository.services.impl.DefaultCleanJobRepository Removing 0 JobInstances
07.04.2016 16:20:01.278 *INFO* [10.0.2.2 [1460046001202] GET /bin/cleanRepository HTTP/1.1] com.twcable.grabbit.spring.batch.repository.services.impl.DefaultCleanJobRepository Removing 0 JobExecutions
07.04.2016 16:20:01.278 *INFO* [10.0.2.2 [1460046001202] GET /bin/cleanRepository HTTP/1.1] com.twcable.grabbit.spring.batch.repository.services.impl.DefaultCleanJobRepository Removing 0 StepExecutions
07.04.2016 16:20:01.278 *INFO* [10.0.2.2 [1460046001202] GET /bin/cleanRepository HTTP/1.1] com.twcable.grabbit.spring.batch.repository.services.impl.DefaultCleanJobRepository Removing 0 JobExecutionContexts
07.04.2016 16:20:01.278 *INFO* [10.0.2.2 [1460046001202] GET /bin/cleanRepository HTTP/1.1] com.twcable.grabbit.spring.batch.repository.services.impl.DefaultCleanJobRepository Removing 0 StepExecutionContexts

You can play around with the "hours" param for the servlet and see how it performs .. deletions will be made based on Comparison between "Date.now() - hours" and "All COMPLETED and FAILED JobExecutions" ... it will also delete all the corresponding "StepExecutions", "JobInstances" and "ExecutionContexts" ...

sagarsane · 2016-04-07T16:29:07Z

...y/com/twcable/grabbit/spring/batch/repository/services/impl/DefaultCleanJobRepository.groovy

+
+         JcrJobRepositoryFactoryBean jobRepositoryFactoryBean = configurableApplicationContext.getBean(JcrJobRepositoryFactoryBean)
+
+         List<String> jobExecutionPaths = jobRepositoryFactoryBean.jobExecutionDao.getJobExecutions([BatchStatus.FAILED, BatchStatus.COMPLETED])


Other "BatchStatus" options are

STARTING, STARTED, STOPPING, STOPPED, ABANDONED, UNKNOWN;

So, should we query for everything except "STARTING" and "STARTED" and "STOPPING" ?

@sagarsane whats the difference between STARTING and STARTED?

From what I understand .. its similar to OSGi's STARTING and ACTIVE bundle states (not 1-1 comparison tho :) )

https://blog.codecentric.de/en/2014/04/spring-batch-batchstatus-state-transitions/ has a bit more info.

sagarsane · 2016-04-11T17:23:32Z

Done with some main CR comments. Still not done anything to suggestions like #107 (comment) .. I like that idea but wanted to discuss that first.

@jazeren1 @jbornemann @jdigger @masroormohammed please take a look .

Dnjoroge · 2016-04-15T16:26:42Z

👍 Testing

jbornemann · 2016-04-18T10:02:38Z

...main/groovy/com/twcable/grabbit/spring/batch/repository/JcrGrabbitExecutionContextDao.groovy

+                Resource jobExecutionResource = resolver.getResource(jobExecutionResourcePath)
+                ValueMap props = jobExecutionResource.adaptTo(ValueMap)
+                Long jobExecutionId = props[JcrGrabbitJobExecutionDao.EXECUTION_ID] as Long
+                String query = "select * from [nt:unstructured] as s " +


s/s/resource/ ?

sagarsane · 2016-04-18T19:10:44Z

...com/twcable/grabbit/spring/batch/repository/servlets/GrabbitCleanJobRepositoryServlet.groovy

+@Slf4j
+@CompileStatic
+@SlingServlet(methods = ['POST'], resourceTypes = ['twcable:grabbit/jobrepository/clean'])
+class GrabbitCleanJobRepositoryServlet extends SlingAllMethodsServlet {


@jbornemann @jazeren1 @jdigger @masroormohammed added this.

POST /grabbit/jobrepository/clean with param 'hours'

I like using a POST API to clean the jobrepository .. that acts as a "resource" .. and that resource manifests with the help of given "hours" param. This servlet returns JobExecutionIds that were removed (if successful)

sagarsane · 2016-04-18T19:14:13Z

Updated testing instructions to have the new API to clean the jobrepository

masroormohammed · 2016-05-06T20:29:12Z

+1 Testing

sagarsane · 2016-06-03T15:36:40Z

CR changes have been made @jdigger @jbornemann

viveksachdeva · 2016-06-27T21:35:35Z

@sagarsane : +1 Testing

…y that is older than X hours from "now" * New API added : POST /grabbit/jobrepository/clean --data hours=X * All JobInstances, JobExecutions, StepExecutions and ExecutionContexts that either FAILED or COMPLETED X hours from "now", i.e., when the API was called -- will be deleted

sagarsane added the in progress label Apr 5, 2016

sagarsane changed the title ~~[WIP] Initial work towards cleaning jobrepository~~ [WIP] Initial work towards automated cleaning of JcrJobRepository Apr 5, 2016

sagarsane force-pushed the 6-clean-jcr-job-repository branch 4 times, most recently from 8d2cb53 to 7d855e2 Compare April 7, 2016 15:58

sagarsane changed the title ~~[WIP] Initial work towards automated cleaning of JcrJobRepository~~ Work towards automated cleaning of JcrJobRepository Apr 7, 2016

sagarsane reviewed Apr 7, 2016
View reviewed changes

jbornemann reviewed Apr 18, 2016
View reviewed changes

sagarsane force-pushed the 6-clean-jcr-job-repository branch from cf69e36 to cf354ce Compare April 18, 2016 19:06

sagarsane reviewed Apr 18, 2016
View reviewed changes

sagarsane force-pushed the 6-clean-jcr-job-repository branch from cf354ce to 90eabe8 Compare April 18, 2016 19:12

sagarsane force-pushed the 6-clean-jcr-job-repository branch from 3e5670c to b39f3b7 Compare June 29, 2016 05:15

sagarsane merged commit b39f3b7 into master Jun 29, 2016

sagarsane deleted the 6-clean-jcr-job-repository branch June 29, 2016 16:36

sagarsane removed the in progress label Jun 29, 2016

sagarsane mentioned this pull request Jun 29, 2016

Implement a way to clean JcrJobRepository data older than X #6

Closed

jdigger mentioned this pull request Jul 18, 2016

Improve status API performance #2 #101

Closed


		JcrJobRepositoryFactoryBean jobRepositoryFactoryBean = configurableApplicationContext.getBean(JcrJobRepositoryFactoryBean)

		List<String> jobExecutionPaths = jobRepositoryFactoryBean.jobExecutionDao.getJobExecutions([BatchStatus.FAILED, BatchStatus.COMPLETED])

Conversation

sagarsane commented Apr 5, 2016

Uh oh!

sagarsane commented Apr 6, 2016

Uh oh!

sagarsane commented Apr 7, 2016

Uh oh!

sagarsane commented Apr 7, 2016

Uh oh!

sagarsane commented Apr 7, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sagarsane Apr 7, 2016

Choose a reason for hiding this comment

Uh oh!

jazeren1 Apr 8, 2016

Choose a reason for hiding this comment

Uh oh!

sagarsane Apr 11, 2016

Choose a reason for hiding this comment

Uh oh!

sagarsane Jun 3, 2016

Choose a reason for hiding this comment

Uh oh!

sagarsane commented Apr 11, 2016

Uh oh!

Dnjoroge commented Apr 15, 2016

Uh oh!

jbornemann Apr 18, 2016

Choose a reason for hiding this comment

Uh oh!

sagarsane Apr 18, 2016

Choose a reason for hiding this comment

Uh oh!

sagarsane commented Apr 18, 2016

Uh oh!

masroormohammed commented May 6, 2016

Uh oh!

sagarsane commented Jun 3, 2016

Uh oh!

viveksachdeva commented Jun 27, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

sagarsane commented Apr 7, 2016 •

edited

Loading

viveksachdeva commented Jun 27, 2016 •

edited

Loading