If there any documentation on the xml schema used within NORMA for ORM schemas?

[AdamBanham]

Hi,

I am building out an editor for a subset of the ORM 2.0 notation. I am looking to support importing an ORM schema into NORMA from my scappy (https://marketplace.visualstudio.com/items?itemName=qORMa.qorma) editor. I was wondering if there was any documentation about xml schema used within NORMA for its file format? Is there minimum amount of information needed to make the file format transferable for elements?

I am not looking to touch on the formal coolness (i.e. verbalisation) that NORMA has built into the studio extension, but instead my editor is going to focus making a nice GUI interface. So having a file format that is compatible with NORMA xml would be helpful.
I am currently using yaml as a temporary file format but a somewhat high-level view of the of the elements I have within the editor currently can be seen in the ORM schema below:

Do you have any idea how feasible it would be to make a compatible XML for NORMA?

Cheers,
Adam

[mcurland]

This turned into a long answer. Please bear with me.

XML Format History/Instability
The NORMA XML format was designed to work with the NORMA loader across multiple extensions. The loading process is standard with xml formats and generators. There are 5 node patterns (root, @id-only, @ref-only, @id and @ref (objectified), no @id or @ref (header elements). In the early NORMA years this was in heavy flux, with each incremental change having accompanying transform(s) to move to the new format. Frankly, it was never meant to be directly read by external programs, but I wasn't able to contain that. The current version has a ton of deprecated elements that are effectively transformed on load and never written out again. I am long overdue for a new file format (same basic pattern, but with changes). A new format would perform upgrades as a transform (not as legacy baggage in the core). However, there are more fundamental things I need to address (for example, data types need a format rework as object types, which allows functions to be replaced with scalar fact types, and sample population needs a rework to enable investigative population before preferred ids are established). There are also legacy issues inherited from file formats 20 years ago (we have always known that orm:Fact is not the right tag for orm:FactType, for example).

XML Schemas
Any data in an .orm file has an associated schema installed with VS. Open the file as XML, go to definition and you'll be in the file. There is one more step in the process for a per-user install (described with the marketplace package instructions). The only slightly non-standard thing I do with the XML is require all namespaces on the root element. The XML namespaces are mapped to extensions, which lets me identify all required extensions before loading the file. (Adding an extension simply puts the namespace on the root element and reloads the file). (There is also a ORM2CoreXsd.htm file that you can download to view. It installed with the pre-vsix setup, but I don't remember where/if it exists now and don't feel like investigating at the moment).

XML Loading
So, basically, heavy reliance on reading/writing the NORMA file format is guaranteed to be a maintenance problem. There are two loaders already (the C# one, and the one in the various file viewers at https://ormsolutions.com/tools/[orm|ormPrint|verbalize]). Basically, I already have a JavaScript loader, as well as a full ORM diagram renderer that looks like ORM. If you use the debugger to drill down into ormViewer.view in any of the viewers you'll see the model and data definitions ready to go (a lot of names are minified, but the data itself is not). There is a lot more than this behind the scenes (save/load to database, import incremental changes, etc.)

JavaScript Environment Editing (Preview)
There are also preliminary editing capabilities here (not publicly posted, but see around 15:48 in the ORM Web Tooling video on ORMSolutions channel. (Sorry, those videos were put up in a hurry and I didn't get them chaptered, or the audio balanced).

Full ORM Editing
There are three major areas with full-fidelity ORM editing in JavaScript:

1. The underlying incremental validation rules are a much larger part of the editing costs than rendering the diagrams. NORMA uses hand-written rules (around 400 of them). Some of the difficulty and complexity results from the DSL framework I built on (problems alleviated by my JavaScript Fact Engine (JFE) framework), but much of it is simply inherent in a large number of complex rules across multiple levels (core, relational, etc.) that all have to work together. Simply put, the dev cost and risk of adding additional rules rises each time something new is added. The only way to sustain this is with a better rule engine and declarative rules (with a generator to determine dependencies and create data for the rule engine). I'm actively doing this right now and plan to get most of it done before fall, but it is obviously a non-trivial project.
2. The NORMA editors (and NORMA Pro rule editors) need to be ported. There are some conceptual components that need to ported. For example, the model browser structure is declarative and built on top of the VS VirtualTreeGrid component, which allows the scaling without performance degradation (and I'm familiar with because I wrote it before leaving Microsoft 2 decades ago). Other pieces (like fact editing) will likely be possible with existing parsing libraries, and will likely be able to leverage existing editors in VSCode for things like the IntelliSense dropdowns.
3. VSCode integration, including artifact and application output. Frankly, this is the area I'm targeting more than anything right now (I have an editor, I don't have a model runner). The goal is to run any model (with UI, derivations, and data validation all live), not just display it.

How to Proceed
It makes a lot more sense to collaborate than trying to remake the world.

• I'm likely many years ahead of you in terms of ORM model analysis, formal ORM model implementation, rich JavaScript data analysis engines, formal rule derivation, etc. I can load .orm files (with a few edge items pending). My ORM SVG rendering is really good. I have a smart data engine that is designed from the ground up to be fact based and mirror an ORM model.
• You're ahead of me in terms of VSCode extension integration and general VSCode experience, including work with existing diagram tech. The diagram-js work also looks good for a start, and is likely better suited to relational rendering and especially line routing (see below) than what I have. A mermaid renderer would be cool to have as well. (Note that the original ORM viewer dates to 2011, well before any of these diagram rendering tools were extant or SVG was consistently available).
• There are a few things neither of us have: direct model execution in VS code (think sample population on steroids with full app validation and object-centric instead of per-fact-type views), relational rendering (I don't have a line routing engine close to what NORMA does, and haven't found one that does properly anchored square-line routing with jumps. The NORMA router is a port of a Visio routing engine from that era-it is far from trivial to emulate). I also need to be able to target more VSCode-friendly data targets (I'm loading/saving through 100% generated server code right now, but the PHP/Doctrine2 stack needs to be targeted to other relational mappers and backends as well).

Bottom Line
I think some of this comes down to motivation for the toolset and long-term (or at least medium term) plans. Over the years, I've lent lots of IP and invested extensive personal time into PhD students working with or adjacent to NORMA. Unfortunately, the efforts tend to vaporize when the colored hoods go on and the research thus far has not resulted in permanent tooling enhancements. Much of the framework (JFE) will be open sourced, but the full tool editing implementation will likely remain private with a public API to access model data.

If you're interested, I'd love to talk. The 16 hours apart is hard (I have young school kids (school ends anyway in 10 days), and school is 1-7am Brisbane, except for Thursdays, where I have until 10am your time). It is easiest to schedule in the next two weeks (Thursday afternoon me/Friday morning you). (Random anecdote: at least we're not trying to schedule Europe/east coast US/east coast Australia/Utah like I've had to do in the past. I invariable pulled the short-straw with a miserable meeting time, like 3am.)

[AdamBanham]

Thanks for the in-depth response. I not was expecting so much details, I was half expecting a "no" as a response. Hopefully, I did not take too much time away from your busy days.

Any data in an .orm file has an associated schema installed with VS. Open the file as XML, go to definition and you'll be in the file. There is one more step in the process for a per-user install (described with the marketplace package instructions). The only slightly non-standard thing I do with the XML is require all namespaces on the root element. The XML namespaces are mapped to extensions, which lets me identify all required extensions before loading the file.

Thanks for the heads up, this is likely where I need to start to make some avenue (however futile) for a compatible data model between the projects/applications.

It is interesting to hear about the development of NORMA over the years. I have mostly been an observer and minor user of the product. At QUT, we do a fair amount of teaching with ORM and after one semester of using draw.io as our diagramer, I had enough. So my scrappy project is hopefully going to avoid the painful experience we had last semester. For many reasons, coordinators are against asking students to install studio over code. Mainly due that in web development, code is more preferred and far less intimidating. This is all to say, in developing my scrappy project I have different goals, but most importantly, the project is not looking to be NORMA. NORMA is NORMA. However, in your response you seem to be interested in something different, and it sounds interesting to me.

My project was mostly inspired from other VS code extensions for conceptual modelling (https://github.com/borkdominik/bigER, https://github.com/borkdominik/bigUML). These show that there are ways to compile server-application into vs-code extensions. Though to me that feels like hitting a nail with a jackhammer, but it has its place. These types of extensions seem to work best with a strong MCV framework in mind, it seems like you editor framework already has this in place.

I think some of this comes down to motivation for the toolset and long-term (or at least medium term) plans. Over the years, I've lent lots of IP and invested extensive personal time into PhD students working with or adjacent to NORMA. Unfortunately, the efforts tend to vaporize when the colored hoods go on and the research thus far has not resulted in permanent tooling enhancements.

Unfortunately, my colored hood is already on. I do understand this perspective though, my aim was not to waste your time by means, not now or in the future. For the most part, this has been a hobby for my own spare time (what little I have) between finding ways to make rent. I will say that I am motivated to see more conceptual modelling and ways to make it easier for the masses (mostly for my teaching purposes).

If you're interested, I'd love to talk. The 16 hours apart is hard (I have young school kids (school ends anyway in 10 days), and school is 1-7am Brisbane, except for Thursdays, where I have until 10am your time). It is easiest to schedule in the next two weeks (Thursday afternoon me/Friday morning you).

I would love to have a discussion and hear about NORMA development. The overlap is rough, but I can make it work. Both early hours on the 23rd and 30th (in Brisbane) would be fine with me. Thankfully, Fridays are my days off from teaching duties. It is probably easier (and safer) to continue the organisation of meeting via email though. If you want organise that meeting, please send me an email to the following address: adam.baham@outlook.com.au adam.banham@outlook.com.au

[AdamBanham]

@mcurland apologies I misspelt my email: adam.banham@outlook.com.au.