Creation of initial decision records. #959

MikeNeilson · 2024-11-22T22:54:15Z

In reference to #942, this is the initial creation of decision records for CDA.

In the current state, as little feedback has been provided on #942 at this time, this is to get feedback on the format of the decision records themselves, but anyone looking at this PR should comment, maybe open PR to this PR to add opinions.

I do not intent to merge these until an actual decision is reached with appropriate feedback; however, barring feedback and discussion such decisions will be made.

MikeNeilson · 2024-11-22T22:56:05Z

I've tagged some specific people for initial review, please feel free to bring in anyone else that may have an opinion. This is a discussion of both the decision record format as well as the actual decisions being recorded.

More will follow, I'm starting small...ish.

MikeNeilson · 2024-11-25T17:46:05Z

@jvanaalsburg considering you started with with cwms-python, probably good to get your feedback about how I'm starting these as well.

docs/decisions/0001-api-versioning.md

adamkorynta · 2025-05-14T17:18:30Z

docs/decisions/0002-data-versioning.md

+
+@MikeNeilson
+
+By versioning the data, and using the Content-Type and Accept headers and the full features of MIME types we appropriately 


I think REST principal accuracy is a good goal, but I think we should consider the following:

API clutter

industry standards

browser support

clear documentation

I would argue that accept header versioning provides the same amount of clutter both in implementation and downstream usage as path parameter versioning. However, it fails at the other three points.

While I would not want to have a tool (Swagger webpage) dictate how we implement an API, I do think it is a non-trivial point when it is our only source of API documentation.
The OpenAPI does not provide a meaningful way to distinguish between different versions on the version marker itself and the Swagger webpage does not make it immediately obvious that there are multplie versions of an endpoint.
Adding/removing/editing behavior of query parameters require extra documentation on the parameter (e.g. "This parameter is not supported by application/json;version=1, application/json;version=2 but is supported by application/json;version=2025-05-01. Now make sure you check what the default is to make sure the parameter is supported by your version."). Why do that to ourselves and users?

Path versioning on the other hand, communicates very clearly to end users what different versions' behaviors will be, both in query parameter behaviors and in what formats are accepted/returned.

Switching to calendar versioning on the accept header gives away the API clutter benefit as now downstream usages will have to manage many different versions beyond easy-to-grok integers, in which case, might as well put the version in the path or as a query parameter to at least provide easier documentation and browser support.

I could see a slight mitigation for the explosion of versions to manage by looping it into the release cycle so they are at least versioned along with the REST API itself, rather than at PR creation date. At that point though, we're just adding more maintenance burden and over time with faster release cycles this becomes moot.

Note: we are expanding the API documentation, Actually I need to move these to rst and that read-the-docs infrastructure.

But I'm still not convinced path versioning is correct. I can see the point about not using dates since they will be somewhat all over the place. But while path version may be an industry standard... it seems a really sloppy industry standard.

Over time, while our outputs have varied slightly, the inputs have only been expanded, not broken in a backwards incompatible way (at least intentionally)

And the few places were they have drastically change, when ended up with a better name of the endpoint anyways, usually something more refined and specific that /cwms-data/v1/locations -> /cwms-data/v2/locations (just for a concrete example).

I have found some arguments and discussion in favor of versioning data, I will take the time to look them up.

Another option, instead of ;version= could just be more expanded data types like application/vnd.json-ts-yada yada (I can't remember off the top of my end the right vnd syntax but I think that gets the idea across)

I'd like to see some examples of how industry does versioning. What is done by Stripe, Twitter, OpenAI etc? Do they version data-types too?

adamkorynta · 2025-05-14T17:20:33Z

docs/decisions/0003-searchability-and-catalogs.md

+
+### Opinion 2
+
+Summary: Each datatype under "catalog" should be a full path"


trailing quote

adamkorynta · 2025-05-14T17:32:55Z

docs/decisions/0003-searchability-and-catalogs.md

+If it makes sense to group all catalogs under catalog, perhaps for grouping in the SWAGGER-UI, making each catalog it's own
+path under `/catalog` instead of the current path parameter is a better approach.
+
+We would maintain the grouping, but each catalog can have it's appropriate search criteria.


This does seem in opposition to the accept header reasoning. We already implement getAll for most (all?) data types. However, most of them return the full data object rather than a listing of refs/ids. The catalog endpoint is supposed to be the lightweight alternative to a getAll, which seems to be a change in the shape of the data. A application/json;catalog=true or something similar.

Fair point, I hadn't thought of just using a different content-type of content-type parameter in that situation.

I will say, while I brought up using the header, I still don't like that solution given my arguments above about discoverability and lack of clear Swagger UI documentation....

In my opinion we haven't done a great job of explaining what a catalog is vs just a bulk-retrieve end-point. I think they are (or can be) different things. In some places we've added flags to enable the controllers to returned reduced data fields or added new *Identifier endpoints but that hasn't been done consistently

In my opinion we haven't done a great job of explaining what a catalog is vs just a bulk-retrieve end-point. I think they are (or can be) different things. In some places we've added flags to enable the controllers to returned reduced data fields or added new *Identifier endpoints but that hasn't been done consistently

We also haven't done a great job of decide if we should even have a "catalog" endpoint or just treat the <dataset>/ as the catalog for each.

MikeNeilson · 2025-10-30T19:01:57Z

NOTE: I will be forcibly merge this in as-is next Friday barring any additional input.

That said this is "initial decision records" should any one decide they think our design goals and intentions are working in the future, you are of course free, and encouraged, to propose changes. But we have to start someone and I don't want this to linger anymore.

docs/source/decisions/0001-api-versioning.rst

adamkorynta · 2025-11-05T22:38:55Z

docs/source/libraries/java.rst

@@ -1,0 +1,3 @@
+####


Is this a decision record for determining which version(s) of java to support?

No, this is a placeholder that was missing from the original sphinx setup that was putting errors in the build output.

docs/source/decisions/index.rst

adamkorynta · 2025-11-05T23:12:00Z

docs/source/decisions/0003-searchability-and-catalogs.rst

+[comment:] <> (Status: request for comments | proposed | accepted | rejected | deprecated | superseded)
+
+References
+==========


recommend adding a resource referencing the law: It's not just a good idea, it's technically the law.

Fair point, that one should actually be that hard to dig up.

References were added.

docs/source/decisions/0003-searchability-and-catalogs.rst

adamkorynta · 2025-11-05T23:29:46Z

docs/source/decisions/0003-searchability-and-catalogs.rst

+
+@MikeNeilson
+
+If it makes sense to group all catalogs under catalog, perhaps for grouping in the SWAGGER-UI, making each catalog it's own


Groups in the swagger-ui are created using the TAG annotation configuration, not by the paths themselves.

I find catalog grouped with the data type easier for discoverability:

timeseries/catalog

timeseries/<name>

location/catalog

location/<name>

It looks like swagger-ui does support multiple tags and allowing the same endpoint to show up under both groups in the UI.

Good to know.

docs/source/decisions/0004-versioning.rst

adamkorynta · 2025-11-05T23:49:17Z

docs/source/decisions/0004-versioning.rst

+
+.. NOTE::
+
+    Or is that confusing and we should just allows add a new endpoint to the highest endpoint version?


Worth calling out how various methods would get versioned with path versioning:

I presume we would want cwms-data/v3/locations GET, cwms-data/v3/locations/<name> GET, cwms-data/v3/locations/<name> DELETE, cwms-data/v3/locations/<name> POST would all get updated to the v3 even if only the cwms-data/v3/locations/<name> GET became backwards incompatible?

Probably best to duplicate

adamkorynta · 2025-11-06T18:16:06Z

docs/source/decisions/0003-searchability-and-catalogs.rst

+to attempt to consildate search query parameters. For TimeSeries and Locations this works reasonably well since there
+is parity between the concepts.
+
+However, if we tried to add ratings into the mix, the list of query parameters grows, and it would rather difficult to 


Can we define what a "catalog" request means under this concept? We already have Javalin CrudHandler getAll implementations under the data types for every data type. There is a redundant /locations getAll already that does similar cataloging as the /catalog/locations. If we are adding an explicit /locations/catalog would there be a difference in implementation? Would the returned results be a smaller payload than the other CRUD endpoints? We would want that to be clear and explicit in this decision doc to avoid confusion.

More questions on this: if we are separating out the CrudHandler getAll endpoints from a /catalog endpoint, would one handle aliases, and would one return a smaller subset of metadata? Ex. /locations/catalog would return office+loc id only but /locations (CrudHandler getAll) would return all physical location metadata and all aliases for each location?

Fair point. "Catalog" here is "I want to search for data", not necessarily I want to get all the data.

My thought behind having <data>/catalog/ vs /catalog/<data> was to try and simplify things; however, since that was written I think we also consider moving the /catalog/<data> to have the <data> part be it's own independent Handler vs a single data type (the amount of query parameters on the /catalog endpoint are getting a bit... nuts.

and having it specifically named "catalog" what to indicate that it's more like a library card catalog than for actual data.

docs/source/decisions/0002-data-versioning.rst

MikeNeilson

Thanks, definitely going to have to consider some of those to fill out some explinations.

MikeNeilson · 2025-11-07T14:39:18Z

docs/source/libraries/java.rst

@@ -1,0 +1,3 @@
+####


No, this is a placeholder that was missing from the original sphinx setup that was putting errors in the build output.

docs/source/decisions/0002-data-versioning.rst

MikeNeilson · 2025-11-07T14:44:43Z

docs/source/decisions/0003-searchability-and-catalogs.rst

+to attempt to consildate search query parameters. For TimeSeries and Locations this works reasonably well since there
+is parity between the concepts.
+
+However, if we tried to add ratings into the mix, the list of query parameters grows, and it would rather difficult to 


Fair point. "Catalog" here is "I want to search for data", not necessarily I want to get all the data.

My thought behind having <data>/catalog/ vs /catalog/<data> was to try and simplify things; however, since that was written I think we also consider moving the /catalog/<data> to have the <data> part be it's own independent Handler vs a single data type (the amount of query parameters on the /catalog endpoint are getting a bit... nuts.

MikeNeilson · 2025-11-07T14:45:01Z

docs/source/decisions/0003-searchability-and-catalogs.rst

+
+@MikeNeilson
+
+If it makes sense to group all catalogs under catalog, perhaps for grouping in the SWAGGER-UI, making each catalog it's own


Good to know.

MikeNeilson · 2025-11-07T14:45:56Z

docs/source/decisions/0003-searchability-and-catalogs.rst

+[comment:] <> (Status: request for comments | proposed | accepted | rejected | deprecated | superseded)
+
+References
+==========


Fair point, that one should actually be that hard to dig up.

MikeNeilson · 2025-11-07T14:46:50Z

docs/source/decisions/0004-versioning.rst

+
+.. NOTE::
+
+    Or is that confusing and we should just allows add a new endpoint to the highest endpoint version?


Probably best to duplicate

MikeNeilson

Updated have been provided based the the given feedback.

docs/source/decisions/0002-data-versioning.rst

adamkorynta · 2025-11-21T19:27:11Z

docs/source/decisions/0003-searchability-and-catalogs.rst

+Catalog - a complete list of items, for examples of things that people can look at or buy [3]
+
+By having a well defined structure of information users can more easily discover what they are looking for. While the
+Sagger-UI, if used, presented all of the types of data that can be found. The `catalog` for each type should present


Suggested change

Sagger-UI, if used, presented all of the types of data that can be found. The `catalog` for each type should present

Swagger-UI, if used, presented all of the types of data that can be found. The `catalog` for each type should present

adamkorynta · 2025-11-21T19:27:34Z

docs/source/decisions/0003-searchability-and-catalogs.rst

+a clear way to find the available data of each type.
+
+The catalog of each data set would include only metadata associated with each data type. For example a time series
+catalog would include support to discover primary timeseries names, aliases, extends, and the like but not actual time


Suggested change

catalog would include support to discover primary timeseries names, aliases, extends, and the like but not actual time

catalog would include support to discover primary timeseries names, aliases, extents, and the like but not actual time

Co-authored-by: Adam Korynta <47677856+adamkorynta@users.noreply.github.com>

MikeNeilson requested review from DanielTOsborne, jbkolze, krowvin and rma-psmorris November 22, 2024 22:54

MikeNeilson added the approved-W192HQ23F0232-task4 Only valid if set by MikeNeilson, DanielO, CharlesG label Dec 5, 2024

MikeNeilson mentioned this pull request Dec 20, 2024

Create a Controller for CDA Server info #515

Open

MikeNeilson mentioned this pull request Mar 7, 2025

Locations Endpoint - Return Aliases #1036

Closed

adamkorynta reviewed May 14, 2025

View reviewed changes

MikeNeilson removed the approved-W192HQ23F0232-task4 Only valid if set by MikeNeilson, DanielO, CharlesG label May 15, 2025

MikeNeilson force-pushed the devops/decision-records branch 2 times, most recently from 4f028a3 to 96f8650 Compare August 4, 2025 14:52

MikeNeilson requested a review from adamkorynta August 4, 2025 14:56

MikeNeilson force-pushed the devops/decision-records branch from 3083c2f to cc24a03 Compare October 30, 2025 19:00

adamkorynta reviewed Nov 5, 2025

View reviewed changes

adamkorynta reviewed Nov 6, 2025

View reviewed changes

MikeNeilson commented Nov 7, 2025

View reviewed changes

MikeNeilson force-pushed the devops/decision-records branch from 424961d to bfb7d7d Compare November 17, 2025 16:13

MikeNeilson commented Nov 17, 2025

View reviewed changes

docs/source/decisions/0002-data-versioning.rst Show resolved Hide resolved

docs/source/decisions/0002-data-versioning.rst Show resolved Hide resolved

adamkorynta previously approved these changes Dec 1, 2025

View reviewed changes

MikeNeilson and others added 7 commits December 3, 2025 17:54

Creation of initial decision records.

9d3212f

Update decision records to rst.

cd055ac

Update language and modified documents to reflect reality.

9ddb27b

Apply suggestions from code review

871a7ce

Co-authored-by: Adam Korynta <47677856+adamkorynta@users.noreply.github.com>

Reponses to feedback.

9085d3b

File correction.

5aca31e

Various updates.

347de19

MikeNeilson dismissed adamkorynta’s stale review via 347de19 December 3, 2025 18:03

MikeNeilson force-pushed the devops/decision-records branch from bfb7d7d to 347de19 Compare December 3, 2025 18:03


		@MikeNeilson

		By versioning the data, and using the Content-Type and Accept headers and the full features of MIME types we appropriately


		### Opinion 2

		Summary: Each datatype under "catalog" should be a full path"


		@MikeNeilson

		If it makes sense to group all catalogs under catalog, perhaps for grouping in the SWAGGER-UI, making each catalog it's own


		.. NOTE::

		Or is that confusing and we should just allows add a new endpoint to the highest endpoint version?

	Sagger-UI, if used, presented all of the types of data that can be found. The `catalog` for each type should present
	Swagger-UI, if used, presented all of the types of data that can be found. The `catalog` for each type should present

	catalog would include support to discover primary timeseries names, aliases, extends, and the like but not actual time
	catalog would include support to discover primary timeseries names, aliases, extents, and the like but not actual time

Creation of initial decision records. #959

Are you sure you want to change the base?

Creation of initial decision records. #959

Uh oh!

Conversation

MikeNeilson commented Nov 22, 2024

Uh oh!

MikeNeilson commented Nov 22, 2024

Uh oh!

MikeNeilson commented Nov 25, 2024

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MikeNeilson commented Oct 30, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MikeNeilson left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!