-
Notifications
You must be signed in to change notification settings - Fork 21
JPERF-273: Jira provisioning hooks #149
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
1e2fb3d to
86ad16e
Compare
317a0d1 to
a141abb
Compare
This reverts commit 71d7136.
Turn `PostgresDatabase` into a hook too.
Include overall Data Center plan with hook timings. Plans should be reusable with different ways of getting a `TcpHost`: * new via Docker * new via AWS * existing machines This commit injects an `Infrastructure` SPI to reuse a plan with different infra, but to prove the concept, it has to fit the current CloudFormation stack in aws-infrastructure / 2-nodes-dc*.yaml The API smells a little, because a big one-shot CFN stack is split into multiple fine-grained methods. Alternative API could be: `fun plan(host: TcpHost) : JiraNodePlan` Other matters to resolve: * loadbalancer (plan?) API * getting results out of DC instance and node hooks
Download reports per host.
2b0843a to
681c0df
Compare
|
Locally failing like this: https://scans.gradle.com/s/375dwevxnzzse |
To read: * https://danwalsh.livejournal.com/74095.html * bmuschko/gradle-docker-plugin#890 Maybe it's because of multiple port exports?
2cd6743 to
3aacf87
Compare
| hooks | ||
| .poll() | ||
| ?.call(ssh, jira, this, reports) | ||
| ?: break |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe this is not important, however I'm gonna note it:
If I'm not wrong it's possible to create a PostInstallHook that'd create an infinite loop here.
| private fun installInParallel(nodes: Collection<PlannedHttpNode>): List<PlannedInstalledJira> = nodes | ||
| .asSequence() | ||
| .asStream() | ||
| .parallel() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are we fine with this parallel stream here?
Avoid errors in Jira logs like: ``` Caused by: java.lang.NullPointerException at sun.awt.FontConfiguration.getVersion(FontConfiguration.java:1264) [?:1.8.0-adoptopenjdk] at sun.awt.FontConfiguration.readFontConfigFile(FontConfiguration.java:219) [?:1.8.0-adoptopenjdk] at sun.awt.FontConfiguration.init(FontConfiguration.java:107) [?:1.8.0-adoptopenjdk] at sun.awt.X11FontManager.createFontConfiguration(X11FontManager.java:774) [?:1.8.0-adoptopenjdk] at sun.font.SunFontManager$2.run(SunFontManager.java:431) [?:1.8.0-adoptopenjdk] at java.security.AccessController.doPrivileged(Native Method) [?:1.8.0-adoptopenjdk] at sun.font.SunFontManager.<init>(SunFontManager.java:376) [?:1.8.0-adoptopenjdk] at sun.awt.FcFontManager.<init>(FcFontManager.java:35) [?:1.8.0-adoptopenjdk] at sun.awt.X11FontManager.<init>(X11FontManager.java:57) [?:1.8.0-adoptopenjdk] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) [?:1.8.0-adoptopenjdk] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) [?:1.8.0-adoptopenjdk] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) [?:1.8.0-adoptopenjdk] at java.lang.reflect.Constructor.newInstance(Constructor.java:423) [?:1.8.0-adoptopenjdk] at java.lang.Class.newInstance(Class.java:442) [?:1.8.0-adoptopenjdk] at sun.font.FontManagerFactory$1.run(FontManagerFactory.java:83) [?:1.8.0-adoptopenjdk] at java.security.AccessController.doPrivileged(Native Method) [?:1.8.0-adoptopenjdk] at sun.font.FontManagerFactory.getInstance(FontManagerFactory.java:74) [?:1.8.0-adoptopenjdk] at java.awt.Font.getFont2D(Font.java:491) [?:1.8.0-adoptopenjdk] at java.awt.Font.canDisplay(Font.java:1980) [?:1.8.0-adoptopenjdk] at com.octo.captcha.component.image.fontgenerator.RandomFontGenerator.checkFontCanDisplayCharacters(RandomFontGenerator.java:223) [jcaptcha-2.0-alpha-1.jar:?] at com.octo.captcha.component.image.fontgenerator.RandomFontGenerator.cleanFontList(RandomFontGenerator.java:179) [jcaptcha-2.0-alpha-1.jar:?] at com.octo.captcha.component.image.fontgenerator.RandomFontGenerator.initializeFonts(RandomFontGenerator.java:94) [jcaptcha-2.0-alpha-1.jar:?] at com.octo.captcha.component.image.fontgenerator.RandomFontGenerator.<init>(RandomFontGenerator.java:81) [jcaptcha-2.0-alpha-1.jar:?] at com.atlassian.jira.servlet.CensoredCaptchaEngine.createFontGenerator(CensoredCaptchaEngine.java:86) [classes/:?] at com.atlassian.jira.servlet.CensoredCaptchaEngine.createWord2Image(CensoredCaptchaEngine.java:65) [classes/:?] at com.atlassian.jira.servlet.CensoredCaptchaEngine.buildInitialFactories(CensoredCaptchaEngine.java:48) [classes/:?] at com.octo.captcha.engine.image.ListImageCaptchaEngine.<init>(ListImageCaptchaEngine.java:24) [jcaptcha-2.0-alpha-1.jar:?] at com.atlassian.jira.servlet.CensoredCaptchaEngine.<init>(CensoredCaptchaEngine.java:39) [classes/:?] at com.atlassian.jira.servlet.JiraImageCaptchaServiceImpl.<clinit>(JiraImageCaptchaServiceImpl.java:13) [classes/:?] ... 137 more ```
Fix `DockerIT`.
Perhaps it will consume less disk space in peaks.
Try to avoid error on GitHub Actions:
```
com.atlassian.performance.tools.infrastructure.api.distribution.PublicJiraServiceDeskDistributionIT > shouldDownloadJiraServiceDesk FAILED
java.lang.Exception at PublicJiraServiceDeskDistributionIT.kt:17
Caused by: net.schmizz.sshj.connection.ConnectionException at PublicJiraServiceDeskDistributionIT.kt:17
Caused by: java.util.concurrent.TimeoutException at PublicJiraServiceDeskDistributionIT.kt:17
java.io.IOException: No space left on device
com.esotericsoftware.kryo.KryoException: java.io.IOException: No space left on device
at com.esotericsoftware.kryo.io.Output.flush(Output.java:165)
```
3aacf87 to
48bb603
Compare
|
|
||
| class Builder( | ||
| private var plan: JiraNodePlan | ||
|
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: empty line
|
|
||
| internal fun call(nodes: List<PreInstallHooks>, reports: Reports) { | ||
| hooks | ||
| .parallelStream() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Another usage of parallel stream. Isn't this risky?
| private fun download(ssh: SshConnection) { | ||
| ssh.execute("sudo mkdir -p $localHome") | ||
| val jiraHome = jiraHomeSource.download(ssh) | ||
| ssh.execute("sudo mv $jiraHome/{data,plugins,import,export} $localHome") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note: Today I noticed that this line will fail when jirahome is empty. I'm writing a test for this in aws-infrastructure with the old API.
In case you'd try to use empty DC instance this may be a problem.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixing it in old API: atlassian/aws-infrastructure#176
Possibly same change should be applied here.
7722512 to
59bceaf
Compare
Tests were consuming too much space in CI, up to ~15 GB in spikes
59bceaf to
c46b1f6
Compare
| import java.time.Duration | ||
| import java.time.Instant | ||
|
|
||
| class JiraPlanIT { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would be good to have also a test for starting empty Jira (no dataset) without setup.
Consumers should provide a working path
Fix errors like:
```
java.lang.Exception: Failed to Installing and configuring apache load balancer despite 2 attempts
at com.atlassian.performance.tools.jvmtasks.api.IdempotentAction.retry(IdempotentAction.kt:42)
at com.atlassian.performance.tools.infrastructure.api.loadbalancer.ApacheProxyPlan.materialize(ApacheProxyPlan.kt:29)
at com.atlassian.performance.tools.infrastructure.api.jira.instance.JiraDataCenterPlan.materialize(JiraDataCenterPlan.kt:30)
at com.atlassian.performance.tools.awsinfrastructure.api.jira.HooksDataCenterFormulaIT$shouldProvisionDc$1.invoke(HooksDataCenterFormulaIT.kt:121)
at com.atlassian.performance.tools.awsinfrastructure.api.jira.HooksDataCenterFormulaIT$shouldProvisionDc$1.invoke(HooksDataCenterFormulaIT.kt:41)
at com.atlassian.performance.tools.awsinfrastructure.api.jira.HooksDataCenterFormulaIT.makeFailureObservable(HooksDataCenterFormulaIT.kt:70)
at com.atlassian.performance.tools.awsinfrastructure.api.jira.HooksDataCenterFormulaIT.shouldProvisionDc(HooksDataCenterFormulaIT.kt:120)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:118)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:175)
at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:157)
at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:404)
at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
at java.lang.Thread.run(Thread.java:750)
Suppressed: java.lang.RuntimeException: Potential Jira problems found in workspace: {}
at com.atlassian.performance.tools.awsinfrastructure.api.jira.HooksDataCenterFormulaIT.makeFailureObservable(HooksDataCenterFormulaIT.kt:74)
... 48 more
Caused by: java.lang.Exception: Error while executing sed -i -r 's/Listen 80/Listen 80/g' /etc/apache2/ports.conf. Exit status code SshResult(exitStatus=4, output=, errorOutput=sed: couldn't open temporary file /etc/apache2/sed3Zlw2Y: Permission denied
)
at com.atlassian.performance.tools.ssh.SshjConnection.execute(SshjConnection.kt:40)
at com.atlassian.performance.tools.ssh.api.SshConnection.execute(SshConnection.kt:27)
at com.atlassian.performance.tools.infrastructure.api.Sed.replace(Sed.kt:14)
at com.atlassian.performance.tools.infrastructure.api.loadbalancer.ApacheProxyPlan.tryToProvision(ApacheProxyPlan.kt:37)
at com.atlassian.performance.tools.infrastructure.api.loadbalancer.ApacheProxyPlan.access$tryToProvision(ApacheProxyPlan.kt:17)
at com.atlassian.performance.tools.infrastructure.api.loadbalancer.ApacheProxyPlan$materialize$1.invoke(ApacheProxyPlan.kt:27)
at com.atlassian.performance.tools.infrastructure.api.loadbalancer.ApacheProxyPlan$materialize$1.invoke(ApacheProxyPlan.kt:17)
at com.atlassian.performance.tools.jvmtasks.api.IdempotentAction.retry(IdempotentAction.kt:28)
... 53 more
```
906087f to
0b3c72c
Compare
1634a16 to
90b1e0e
Compare
90b1e0e to
cacb5f0
Compare

This is the 2023 version of #103
Rebased, resolved a lot of conflicts and cleaned up a little. Turns out many changes were done independently on
masteralready. Still gotta squash them commits...