Allow publishing of IESI script executions to GCP PubSub by BerrevoetsRobbe · Pull Request #182 · metadew/iesi

BerrevoetsRobbe · 2020-08-31T09:09:14Z

Is your feature request related to a problem? Please describe.
I want to be able to configure a GCP PubSub where IESI should publish all the run ids of scripts that have finished.

Id
4fb0d16

…script-executions-to-gcp-pubsub

peterbillen

add empty conf section to /core/conf/application.yml
publishers:
- type:
  project:
  topic:
remove the scriptresult object from the console output and replace the object just by the word 'scriptresult'

peterbillen · 2020-09-11T12:27:27Z

@BerrevoetsRobbe How to deserialize a string to the ScriptResult object?

I am getting a nullpointer exception here

java.lang.NullPointerException
	at io.metadew.iesi.metadata.definition.MetadataJsonComponent$Deserializer.deserialize(MetadataJsonComponent.java:43)
	at io.metadew.iesi.metadata.definition.MetadataJsonComponent$Deserializer.deserialize(MetadataJsonComponent.java:38)

peterbillen · 2020-09-11T12:34:57Z

Additionally, I am getting the following test errors on the branch:

[ERROR] Tests run: 16, Failures: 0, Errors: 5, Skipped: 0, Time elapsed: 44.765 s <<< FAILURE! - in io.metadew.iesi.metadata.configuration.component.ComponentAttributeConfigurationTest
[ERROR] io.metadew.iesi.metadata.configuration.component.ComponentAttributeConfigurationTest.componentAttributeDeleteTest  Time elapsed: 2.567 s  <<< ERROR!
java.lang.RuntimeException: org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (unrecognized token: "1AND")
	at io.metadew.iesi.metadata.configuration.component.ComponentAttributeConfigurationTest.componentAttributeDeleteTest(ComponentAttributeConfigurationTest.java:115)
Caused by: org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (unrecognized token: "1AND")
	at io.metadew.iesi.metadata.configuration.component.ComponentAttributeConfigurationTest.componentAttributeDeleteTest(ComponentAttributeConfigurationTest.java:115)

[ERROR] io.metadew.iesi.metadata.configuration.component.ComponentAttributeConfigurationTest.componentAttributeUpdate1Test  Time elapsed: 2.967 s  <<< ERROR!
java.lang.RuntimeException: org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (unrecognized token: "1AND")
	at io.metadew.iesi.metadata.configuration.component.ComponentAttributeConfigurationTest.componentAttributeUpdate1Test(ComponentAttributeConfigurationTest.java:163)
Caused by: org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (unrecognized token: "1AND")
	at io.metadew.iesi.metadata.configuration.component.ComponentAttributeConfigurationTest.componentAttributeUpdate1Test(ComponentAttributeConfigurationTest.java:163)

[ERROR] io.metadew.iesi.metadata.configuration.component.ComponentAttributeConfigurationTest.componentAttributeUpdate2Test  Time elapsed: 3.038 s  <<< ERROR!
java.lang.RuntimeException: org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (unrecognized token: "1AND")
	at io.metadew.iesi.metadata.configuration.component.ComponentAttributeConfigurationTest.componentAttributeUpdate2Test(ComponentAttributeConfigurationTest.java:218)
Caused by: org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (unrecognized token: "1AND")
	at io.metadew.iesi.metadata.configuration.component.ComponentAttributeConfigurationTest.componentAttributeUpdate2Test(ComponentAttributeConfigurationTest.java:218)

[ERROR] io.metadew.iesi.metadata.configuration.component.ComponentAttributeConfigurationTest.componentAttributeUpdate3Test  Time elapsed: 3.141 s  <<< ERROR!
java.lang.RuntimeException: org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (unrecognized token: "2AND")
	at io.metadew.iesi.metadata.configuration.component.ComponentAttributeConfigurationTest.componentAttributeUpdate3Test(ComponentAttributeConfigurationTest.java:273)
Caused by: org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (unrecognized token: "2AND")
	at io.metadew.iesi.metadata.configuration.component.ComponentAttributeConfigurationTest.componentAttributeUpdate3Test(ComponentAttributeConfigurationTest.java:273)

[ERROR] io.metadew.iesi.metadata.configuration.component.ComponentAttributeConfigurationTest.componentAttributeUpdate4Test  Time elapsed: 2.959 s  <<< ERROR!
java.lang.RuntimeException: org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (unrecognized token: "1AND")
	at io.metadew.iesi.metadata.configuration.component.ComponentAttributeConfigurationTest.componentAttributeUpdate4Test(ComponentAttributeConfigurationTest.java:329)
Caused by: org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (unrecognized token: "1AND")
	at io.metadew.iesi.metadata.configuration.component.ComponentAttributeConfigurationTest.componentAttributeUpdate4Test(ComponentAttributeConfigurationTest.java:329)

[INFO] Running io.metadew.iesi.metadata.configuration.component.ComponentVersionConfigurationTest

BerrevoetsRobbe · 2020-09-11T13:35:55Z

@BerrevoetsRobbe How to deserialize a string to the ScriptResult object?

I am getting a nullpointer exception here

java.lang.NullPointerException
	at io.metadew.iesi.metadata.definition.MetadataJsonComponent$Deserializer.deserialize(MetadataJsonComponent.java:43)
	at io.metadew.iesi.metadata.definition.MetadataJsonComponent$Deserializer.deserialize(MetadataJsonComponent.java:38)

What code are you executing? It seems that you are mapping to Metadata.class and not ScriptResult.claa

BerrevoetsRobbe · 2020-09-11T13:52:13Z

Additionally, I am getting the following test errors on the branch:

[ERROR] Tests run: 16, Failures: 0, Errors: 5, Skipped: 0, Time elapsed: 44.765 s <<< FAILURE! - in io.metadew.iesi.metadata.configuration.component.ComponentAttributeConfigurationTest
[ERROR] io.metadew.iesi.metadata.configuration.component.ComponentAttributeConfigurationTest.componentAttributeDeleteTest  Time elapsed: 2.567 s  <<< ERROR!
java.lang.RuntimeException: org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (unrecognized token: "1AND")
	at io.metadew.iesi.metadata.configuration.component.ComponentAttributeConfigurationTest.componentAttributeDeleteTest(ComponentAttributeConfigurationTest.java:115)
Caused by: org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (unrecognized token: "1AND")
	at io.metadew.iesi.metadata.configuration.component.ComponentAttributeConfigurationTest.componentAttributeDeleteTest(ComponentAttributeConfigurationTest.java:115)

[ERROR] io.metadew.iesi.metadata.configuration.component.ComponentAttributeConfigurationTest.componentAttributeUpdate1Test  Time elapsed: 2.967 s  <<< ERROR!
java.lang.RuntimeException: org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (unrecognized token: "1AND")
	at io.metadew.iesi.metadata.configuration.component.ComponentAttributeConfigurationTest.componentAttributeUpdate1Test(ComponentAttributeConfigurationTest.java:163)
Caused by: org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (unrecognized token: "1AND")
	at io.metadew.iesi.metadata.configuration.component.ComponentAttributeConfigurationTest.componentAttributeUpdate1Test(ComponentAttributeConfigurationTest.java:163)

[ERROR] io.metadew.iesi.metadata.configuration.component.ComponentAttributeConfigurationTest.componentAttributeUpdate2Test  Time elapsed: 3.038 s  <<< ERROR!
java.lang.RuntimeException: org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (unrecognized token: "1AND")
	at io.metadew.iesi.metadata.configuration.component.ComponentAttributeConfigurationTest.componentAttributeUpdate2Test(ComponentAttributeConfigurationTest.java:218)
Caused by: org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (unrecognized token: "1AND")
	at io.metadew.iesi.metadata.configuration.component.ComponentAttributeConfigurationTest.componentAttributeUpdate2Test(ComponentAttributeConfigurationTest.java:218)

[ERROR] io.metadew.iesi.metadata.configuration.component.ComponentAttributeConfigurationTest.componentAttributeUpdate3Test  Time elapsed: 3.141 s  <<< ERROR!
java.lang.RuntimeException: org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (unrecognized token: "2AND")
	at io.metadew.iesi.metadata.configuration.component.ComponentAttributeConfigurationTest.componentAttributeUpdate3Test(ComponentAttributeConfigurationTest.java:273)
Caused by: org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (unrecognized token: "2AND")
	at io.metadew.iesi.metadata.configuration.component.ComponentAttributeConfigurationTest.componentAttributeUpdate3Test(ComponentAttributeConfigurationTest.java:273)

[ERROR] io.metadew.iesi.metadata.configuration.component.ComponentAttributeConfigurationTest.componentAttributeUpdate4Test  Time elapsed: 2.959 s  <<< ERROR!
java.lang.RuntimeException: org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (unrecognized token: "1AND")
	at io.metadew.iesi.metadata.configuration.component.ComponentAttributeConfigurationTest.componentAttributeUpdate4Test(ComponentAttributeConfigurationTest.java:329)
Caused by: org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (unrecognized token: "1AND")
	at io.metadew.iesi.metadata.configuration.component.ComponentAttributeConfigurationTest.componentAttributeUpdate4Test(ComponentAttributeConfigurationTest.java:329)

[INFO] Running io.metadew.iesi.metadata.configuration.component.ComponentVersionConfigurationTest

Not reproducable on my machine. All tests succeed and

Additionally, I am getting the following test errors on the branch:

[ERROR] Tests run: 16, Failures: 0, Errors: 5, Skipped: 0, Time elapsed: 44.765 s <<< FAILURE! - in io.metadew.iesi.metadata.configuration.component.ComponentAttributeConfigurationTest
[ERROR] io.metadew.iesi.metadata.configuration.component.ComponentAttributeConfigurationTest.componentAttributeDeleteTest  Time elapsed: 2.567 s  <<< ERROR!
java.lang.RuntimeException: org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (unrecognized token: "1AND")
	at io.metadew.iesi.metadata.configuration.component.ComponentAttributeConfigurationTest.componentAttributeDeleteTest(ComponentAttributeConfigurationTest.java:115)
Caused by: org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (unrecognized token: "1AND")
	at io.metadew.iesi.metadata.configuration.component.ComponentAttributeConfigurationTest.componentAttributeDeleteTest(ComponentAttributeConfigurationTest.java:115)

[ERROR] io.metadew.iesi.metadata.configuration.component.ComponentAttributeConfigurationTest.componentAttributeUpdate1Test  Time elapsed: 2.967 s  <<< ERROR!
java.lang.RuntimeException: org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (unrecognized token: "1AND")
	at io.metadew.iesi.metadata.configuration.component.ComponentAttributeConfigurationTest.componentAttributeUpdate1Test(ComponentAttributeConfigurationTest.java:163)
Caused by: org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (unrecognized token: "1AND")
	at io.metadew.iesi.metadata.configuration.component.ComponentAttributeConfigurationTest.componentAttributeUpdate1Test(ComponentAttributeConfigurationTest.java:163)

[ERROR] io.metadew.iesi.metadata.configuration.component.ComponentAttributeConfigurationTest.componentAttributeUpdate2Test  Time elapsed: 3.038 s  <<< ERROR!
java.lang.RuntimeException: org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (unrecognized token: "1AND")
	at io.metadew.iesi.metadata.configuration.component.ComponentAttributeConfigurationTest.componentAttributeUpdate2Test(ComponentAttributeConfigurationTest.java:218)
Caused by: org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (unrecognized token: "1AND")
	at io.metadew.iesi.metadata.configuration.component.ComponentAttributeConfigurationTest.componentAttributeUpdate2Test(ComponentAttributeConfigurationTest.java:218)

[ERROR] io.metadew.iesi.metadata.configuration.component.ComponentAttributeConfigurationTest.componentAttributeUpdate3Test  Time elapsed: 3.141 s  <<< ERROR!
java.lang.RuntimeException: org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (unrecognized token: "2AND")
	at io.metadew.iesi.metadata.configuration.component.ComponentAttributeConfigurationTest.componentAttributeUpdate3Test(ComponentAttributeConfigurationTest.java:273)
Caused by: org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (unrecognized token: "2AND")
	at io.metadew.iesi.metadata.configuration.component.ComponentAttributeConfigurationTest.componentAttributeUpdate3Test(ComponentAttributeConfigurationTest.java:273)

[ERROR] io.metadew.iesi.metadata.configuration.component.ComponentAttributeConfigurationTest.componentAttributeUpdate4Test  Time elapsed: 2.959 s  <<< ERROR!
java.lang.RuntimeException: org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (unrecognized token: "1AND")
	at io.metadew.iesi.metadata.configuration.component.ComponentAttributeConfigurationTest.componentAttributeUpdate4Test(ComponentAttributeConfigurationTest.java:329)
Caused by: org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (unrecognized token: "1AND")
	at io.metadew.iesi.metadata.configuration.component.ComponentAttributeConfigurationTest.componentAttributeUpdate4Test(ComponentAttributeConfigurationTest.java:329)

[INFO] Running io.metadew.iesi.metadata.configuration.component.ComponentVersionConfigurationTest

Should be fixed with latests commit. Currently a H2 in-memory is used as db during unit tests, not a SQLite DB.
It seems that H2 is more resilient against than SQLite.

peterbillen · 2020-09-11T14:45:04Z

@BerrevoetsRobbe How to deserialize a string to the ScriptResult object?
I am getting a nullpointer exception here
java.lang.NullPointerException
	at io.metadew.iesi.metadata.definition.MetadataJsonComponent$Deserializer.deserialize(MetadataJsonComponent.java:43)
	at io.metadew.iesi.metadata.definition.MetadataJsonComponent$Deserializer.deserialize(MetadataJsonComponent.java:38)
What code are you executing? It seems that you are mapping to Metadata.class and not ScriptResult.claa

import com.fasterxml.jackson.databind.ObjectMapper;
import io.metadew.iesi.metadata.definition.script.result.ScriptResult;

public class Test {
    public static void main( String[] args ) {
        System.out.println("test");
        ObjectMapper objectMapper = new ObjectMapper();
            String data = "{\"metadataKey\":{\"runId\":\"0f460498-e1cd-4a2d-b914-058fbf9a8b9a\",\"processId\":-1},\"parentProcessId\":-1,\"scriptId\":\"936a185caaa266bb9cbe981e9e05cb78cd732b0b3280eb944412bb6f8f8f07af\",\"scriptName\":\"helloworld\",\"scriptVersion\":0,\"environment\":\"tutorial\",\"status\":\"SUCCESS\",\"startTimestamp\":{\"year\":2020,\"month\":\"SEPTEMBER\",\"dayOfMonth\":11,\"dayOfWeek\":\"FRIDAY\",\"dayOfYear\":255,\"monthValue\":9,\"nano\":985000000,\"hour\":13,\"minute\":29,\"second\":20,\"chronology\":{\"calendarType\":\"iso8601\",\"id\":\"ISO\"}},\"endTimestamp\":{\"year\":2020,\"month\":\"SEPTEMBER\",\"dayOfMonth\":11,\"dayOfWeek\":\"FRIDAY\",\"dayOfYear\":255,\"monthValue\":9,\"nano\":690000000,\"hour\":13,\"minute\":29,\"second\":21,\"chronology\":{\"calendarType\":\"iso8601\",\"id\":\"ISO\"}}}";
        try {
            ScriptResult scriptResult = objectMapper.readValue(data, ScriptResult.class);
            System.out.println(scriptResult);
        } catch (Exception e) {
            e.printStackTrace();
            System.out.println("issue converting object");
        }
    }

}

BerrevoetsRobbe · 2020-09-13T02:05:41Z

@BerrevoetsRobbe How to deserialize a string to the ScriptResult object?
I am getting a nullpointer exception here
java.lang.NullPointerException
	at io.metadew.iesi.metadata.definition.MetadataJsonComponent$Deserializer.deserialize(MetadataJsonComponent.java:43)
	at io.metadew.iesi.metadata.definition.MetadataJsonComponent$Deserializer.deserialize(MetadataJsonComponent.java:38)
What code are you executing? It seems that you are mapping to Metadata.class and not ScriptResult.claa

import com.fasterxml.jackson.databind.ObjectMapper;
import io.metadew.iesi.metadata.definition.script.result.ScriptResult;

public class Test {
    public static void main( String[] args ) {
        System.out.println("test");
        ObjectMapper objectMapper = new ObjectMapper();
            String data = "{\"metadataKey\":{\"runId\":\"0f460498-e1cd-4a2d-b914-058fbf9a8b9a\",\"processId\":-1},\"parentProcessId\":-1,\"scriptId\":\"936a185caaa266bb9cbe981e9e05cb78cd732b0b3280eb944412bb6f8f8f07af\",\"scriptName\":\"helloworld\",\"scriptVersion\":0,\"environment\":\"tutorial\",\"status\":\"SUCCESS\",\"startTimestamp\":{\"year\":2020,\"month\":\"SEPTEMBER\",\"dayOfMonth\":11,\"dayOfWeek\":\"FRIDAY\",\"dayOfYear\":255,\"monthValue\":9,\"nano\":985000000,\"hour\":13,\"minute\":29,\"second\":20,\"chronology\":{\"calendarType\":\"iso8601\",\"id\":\"ISO\"}},\"endTimestamp\":{\"year\":2020,\"month\":\"SEPTEMBER\",\"dayOfMonth\":11,\"dayOfWeek\":\"FRIDAY\",\"dayOfYear\":255,\"monthValue\":9,\"nano\":690000000,\"hour\":13,\"minute\":29,\"second\":21,\"chronology\":{\"calendarType\":\"iso8601\",\"id\":\"ISO\"}}}";
        try {
            ScriptResult scriptResult = objectMapper.readValue(data, ScriptResult.class);
            System.out.println(scriptResult);
        } catch (Exception e) {
            e.printStackTrace();
            System.out.println("issue converting object");
        }
    }

}

I've modified the Java object being sent to io.metadew.iesi.connection.publisher.ScriptResultDto. You can see how it's serialized/deserialized at io.metadew.iesi.connection.publisher.ScriptResultDtoService/io.metadew.iesi.connection.publisher.ScriptResultDtoServiceTest

…script-executions-to-gcp-pubsub

BerrevoetsRobbe added the enhancement New feature or request label Aug 3, 2020

BerrevoetsRobbe added this to the v0.6.0 milestone Aug 3, 2020

BerrevoetsRobbe added 3 commits August 14, 2020 23:42

add basic classes

819eb4e

Merge branch 'develop' into feature/4fb0d16/allow-publishing-of-iesi-…

8323db6

…script-executions-to-gcp-pubsub

add custom json deserializer

f0d64d7

BerrevoetsRobbe self-assigned this Aug 24, 2020

BerrevoetsRobbe added 3 commits August 24, 2020 14:20

Merge branch 'develop' into feature/4fb0d16/allow-publishing-of-iesi-…

3820237

…script-executions-to-gcp-pubsub

clean up

6b88958

Merge branch 'develop' into feature/4fb0d16/allow-publishing-of-iesi-…

259905f

…script-executions-to-gcp-pubsub

BerrevoetsRobbe requested review from louis878 and peterbillen August 31, 2020 13:02

peterbillen reviewed Sep 2, 2020

View reviewed changes

fix delete statement of component attribute

62628c2

BerrevoetsRobbe added 2 commits September 13, 2020 03:17

modify script result dto object

6f620b8

refactor script result dto object

4675cc6

Merge branch 'develop' into feature/4fb0d16/allow-publishing-of-iesi-…

71c4801

…script-executions-to-gcp-pubsub

BerrevoetsRobbe modified the milestones: v0.6.0, v0.7.0 Mar 5, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comments

Allow publishing of IESI script executions to GCP PubSub#182

Allow publishing of IESI script executions to GCP PubSub#182
BerrevoetsRobbe wants to merge 10 commits intodevelopfrom
feature/4fb0d16/allow-publishing-of-iesi-script-executions-to-gcp-pubsub

BerrevoetsRobbe commented Aug 31, 2020 •

edited

Loading

Uh oh!

peterbillen left a comment

Uh oh!

peterbillen commented Sep 11, 2020

Uh oh!

peterbillen commented Sep 11, 2020

Uh oh!

BerrevoetsRobbe commented Sep 11, 2020

Uh oh!

BerrevoetsRobbe commented Sep 11, 2020

Uh oh!

peterbillen commented Sep 11, 2020 •

edited

Loading

Uh oh!

BerrevoetsRobbe commented Sep 13, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Comments

Conversation

BerrevoetsRobbe commented Aug 31, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

peterbillen left a comment

Choose a reason for hiding this comment

Uh oh!

peterbillen commented Sep 11, 2020

Uh oh!

peterbillen commented Sep 11, 2020

Uh oh!

BerrevoetsRobbe commented Sep 11, 2020

Uh oh!

BerrevoetsRobbe commented Sep 11, 2020

Uh oh!

peterbillen commented Sep 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

BerrevoetsRobbe commented Sep 13, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

BerrevoetsRobbe commented Aug 31, 2020 •

edited

Loading

peterbillen commented Sep 11, 2020 •

edited

Loading