From 9d3212f043c6e2f9fea72d21f07666cb78ad75e9 Mon Sep 17 00:00:00 2001 From: Mike Neilson Date: Fri, 22 Nov 2024 22:51:09 +0000 Subject: [PATCH 1/7] Creation of initial decision records. --- docs/decisions/0001-api-versioning.md | 44 +++++++++++++++ docs/decisions/0002-data-versioning.md | 55 +++++++++++++++++++ .../0003-searchability-and-catalogs.md | 42 ++++++++++++++ docs/decisions/adr-template.md | 19 +++++++ 4 files changed, 160 insertions(+) create mode 100644 docs/decisions/0001-api-versioning.md create mode 100644 docs/decisions/0002-data-versioning.md create mode 100644 docs/decisions/0003-searchability-and-catalogs.md create mode 100644 docs/decisions/adr-template.md diff --git a/docs/decisions/0001-api-versioning.md b/docs/decisions/0001-api-versioning.md new file mode 100644 index 000000000..1cbac020c --- /dev/null +++ b/docs/decisions/0001-api-versioning.md @@ -0,0 +1,44 @@ +# API Versioning and Reporting + +## Summary + +API should provide a reason version and list/matrix of capabilities for a given instance of CDA + +## Opinions + +### Opinion 1 Calendar versioning + +@MikeNeilson + +Summary: Calendar versioning is easier to support and automate with this API + +As the one whose has been making the "official" releases Semantic Version versioning has basically been useless. +We have been making so many feature additions that if I was doing it right we'd never have a minor version change between releases. +It's also caused me to, I think, release too slowly. + +While we can automate SemVer it is an additional step. + +With Calendar Versioning automation tools can just pick the current date when appropriately triggered, +perhaps by merged into a particular branch. + + +### Opinion 2 Users + +Summary: It has been asked more than once that a version be provided + +Having a version allows client to better respond to what's available instead of failing in obtuse ways. + + +## Decision Status + +accepted + +1. Provide endpoint to retrieve current API version. +2. Likely include capability list or matrix. + +## References + + +## Related decisions + +- data-versioning diff --git a/docs/decisions/0002-data-versioning.md b/docs/decisions/0002-data-versioning.md new file mode 100644 index 000000000..5a744a427 --- /dev/null +++ b/docs/decisions/0002-data-versioning.md @@ -0,0 +1,55 @@ +# Data Types use Calendar Versioning + +## Summary + +Instead of versioning the entire API, though the API has a version, we version the data types. + +## Opinions + +### Opinion 1 + +Summary: Versioning the API itself, at the level of the path, will lead to too many paths to manage and be awkward for the clients + +@MikeNeilson + +By versioning the data, and using the Content-Type and Accept headers and the full features of MIME types we appropriately +separate the concern of "what data we are retrieving/storing" from the presentation of data. + +e.g /timeseries/Alder Springs.Temp-Air.Inst.15Minutes.0.GOES-raw?begin=PT0&end=PT-1D&units=C, granted with a reasonable + exception to the units, defines *what* we want. + +The header, Accept, informs the API what format, or formats, we are willing to accept the data in. + + + + +## Decision Status + +proposed + +Data, by content-types, are versioned. In the past there was some severe confusion on this part and it was treated as anything new was "version=2" in the content-type. To allow this design but reduce confusion going forward + +1. The initial version of a data set *SHALL* be the date it was finalized (e.g. PR about to be merged.) +2. *IF* it is the first version of this data the plain content-type "e.g. application/json" will point to this data. +3. *IF* is it not the version version of this data, it will be discussed and announced when it becomes the new default data. +4. Downstream systems *SHOULD* use the specific version regardless of when implemented, and this behavior should be well documented. +5. If a given data set includes definitions of its shape within the type there should be sufficient documentation for downstream + developers to properly account for any changes over time. (See our TimeSeries type and discussions within #927). + +Version format is `YYYY-MM-DD` + +[comment:] <> (Status: request for comments | proposed | accepted | rejected | deprecated | superseded) + +## References + +I have several, I will dig them up likely next week + +## Notes + +The initial idea in CDA was that the first version of any data type was, we'll just stick with JSON for each of message, +"application/json;version=1" with "application/json" being the alias to the latest format version. However, this was not +correctly communicated and several brand new data transfer objects were created as ";version=2" under the impression that +this was the version for the new system. Attempting to use a simple number of this has clearly caused confusion in general. + +We also failed to create the initial alias system which caused even more confusion when users attempted to test things +directly in a browser instead of the provided swagger-ui. diff --git a/docs/decisions/0003-searchability-and-catalogs.md b/docs/decisions/0003-searchability-and-catalogs.md new file mode 100644 index 000000000..6e1d33992 --- /dev/null +++ b/docs/decisions/0003-searchability-and-catalogs.md @@ -0,0 +1,42 @@ +# Data should be readily searchable + +## Summary + +It's not just a good idea, it's technically the law. While CDA currently expose a fair amount of information to search +it's never entirely clear. + +## Opinions + +### Opinion 1 + +Summary: Each data time should support it's own /catalog end point. + +@MikeNeilson + +The original CDA has the say, `/timeseries` end point provide a catalog if no data is set. I created a /catalog end point +to attempt to consildate search query parameters. For TimeSeries and Locations this works reasonably well since there +is parity between the concepts. + +However, if we tried to add ratings into the mix, the list of query parameters grows, and it would rather difficult to +document which is for what or what changes for each. + +To make 'catalog' operations clear, we should create /catalog for each data type that provide for discoverability of that data. + +### Opinion 2 + +Summary: Each datatype under "catalog" should be a full path" + +@MikeNeilson + +If it makes sense to group all catalogs under catalog, perhaps for grouping in the SWAGGER-UI, making each catalog it's own +path under `/catalog` instead of the current path parameter is a better approach. + +We would maintain the grouping, but each catalog can have it's appropriate search criteria. + +## Decision Status + +proposed + +[comment:] <> (Status: request for comments | proposed | accepted | rejected | deprecated | superseded) + +## References diff --git a/docs/decisions/adr-template.md b/docs/decisions/adr-template.md new file mode 100644 index 000000000..6fad01441 --- /dev/null +++ b/docs/decisions/adr-template.md @@ -0,0 +1,19 @@ +# Title + +## Summary + +## Opinions + +### Opinion 1 + +Summary: + +Author + +descriptive text + +## Decision Status + +[comment:] <> (Status: request for comments | proposed | accepted | rejected | deprecated | superseded) + +## References From cd055ac879bd54407c385f25e893415cc15c37f1 Mon Sep 17 00:00:00 2001 From: Mike Neilson Date: Mon, 4 Aug 2025 14:47:35 +0000 Subject: [PATCH 2/7] Update decision records to rst. --- docs/source/data/index.rst | 15 +++++ .../decisions/0001-api-versioning.rst} | 25 +++++--- .../decisions/0002-data-versioning.rst} | 24 +++++--- .../0003-searchability-and-catalogs.rst} | 22 ++++--- docs/source/decisions/0004-versioning.rst | 61 +++++++++++++++++++ .../decisions/adr-template.rst} | 20 ++++-- docs/source/decisions/index.rst | 23 +++++++ docs/source/index.rst | 12 +++- docs/source/libraries/java.rst | 2 - 9 files changed, 170 insertions(+), 34 deletions(-) rename docs/{decisions/0001-api-versioning.md => source/decisions/0001-api-versioning.rst} (76%) rename docs/{decisions/0002-data-versioning.md => source/decisions/0002-data-versioning.rst} (91%) rename docs/{decisions/0003-searchability-and-catalogs.md => source/decisions/0003-searchability-and-catalogs.rst} (85%) create mode 100644 docs/source/decisions/0004-versioning.rst rename docs/{decisions/adr-template.md => source/decisions/adr-template.rst} (53%) create mode 100644 docs/source/decisions/index.rst diff --git a/docs/source/data/index.rst b/docs/source/data/index.rst index f8a53e2f0..a01531ddc 100644 --- a/docs/source/data/index.rst +++ b/docs/source/data/index.rst @@ -1,3 +1,4 @@ +<<<<<<< HEAD Data Overview ================ @@ -7,3 +8,17 @@ Data Overview Timeseries <./timeseries.rst> Location Levels <./location-levels.rst> +======= +#### +Data +#### + + +.. toctree: + :maxdepth: 1 + :caption: Data + + + Location Levels <./location-levels.rst> + Time Series <./timeseries.rst> +>>>>>>> 2fcd1efd (Update decision records to rst.) diff --git a/docs/decisions/0001-api-versioning.md b/docs/source/decisions/0001-api-versioning.rst similarity index 76% rename from docs/decisions/0001-api-versioning.md rename to docs/source/decisions/0001-api-versioning.rst index 1cbac020c..420c5ac74 100644 --- a/docs/decisions/0001-api-versioning.md +++ b/docs/source/decisions/0001-api-versioning.rst @@ -1,12 +1,17 @@ -# API Versioning and Reporting +############################ +API Versioning and Reporting +############################ -## Summary +Summary +======= API should provide a reason version and list/matrix of capabilities for a given instance of CDA -## Opinions +Opinions +======== -### Opinion 1 Calendar versioning +Opinion 1 Calendar versioning +----------------------------- @MikeNeilson @@ -22,23 +27,27 @@ With Calendar Versioning automation tools can just pick the current date when ap perhaps by merged into a particular branch. -### Opinion 2 Users +Opinion 2 Users +--------------- Summary: It has been asked more than once that a version be provided Having a version allows client to better respond to what's available instead of failing in obtuse ways. -## Decision Status +Decision Status +=============== accepted 1. Provide endpoint to retrieve current API version. 2. Likely include capability list or matrix. -## References +References +========== -## Related decisions +Related decisions +================= - data-versioning diff --git a/docs/decisions/0002-data-versioning.md b/docs/source/decisions/0002-data-versioning.rst similarity index 91% rename from docs/decisions/0002-data-versioning.md rename to docs/source/decisions/0002-data-versioning.rst index 5a744a427..d5e28ad4d 100644 --- a/docs/decisions/0002-data-versioning.md +++ b/docs/source/decisions/0002-data-versioning.rst @@ -1,12 +1,17 @@ -# Data Types use Calendar Versioning +################################## +Data Types use Calendar Versioning +################################## -## Summary +Summary +======= Instead of versioning the entire API, though the API has a version, we version the data types. -## Opinions +Opinions +======== -### Opinion 1 +Opinion 1 +--------- Summary: Versioning the API itself, at the level of the path, will lead to too many paths to manage and be awkward for the clients @@ -23,9 +28,10 @@ The header, Accept, informs the API what format, or formats, we are willing to a -## Decision Status +Decision Status +--------------- -proposed +partial acceptance Data, by content-types, are versioned. In the past there was some severe confusion on this part and it was treated as anything new was "version=2" in the content-type. To allow this design but reduce confusion going forward @@ -40,11 +46,13 @@ Version format is `YYYY-MM-DD` [comment:] <> (Status: request for comments | proposed | accepted | rejected | deprecated | superseded) -## References +References +========== I have several, I will dig them up likely next week -## Notes +Notes +===== The initial idea in CDA was that the first version of any data type was, we'll just stick with JSON for each of message, "application/json;version=1" with "application/json" being the alias to the latest format version. However, this was not diff --git a/docs/decisions/0003-searchability-and-catalogs.md b/docs/source/decisions/0003-searchability-and-catalogs.rst similarity index 85% rename from docs/decisions/0003-searchability-and-catalogs.md rename to docs/source/decisions/0003-searchability-and-catalogs.rst index 6e1d33992..62996ea70 100644 --- a/docs/decisions/0003-searchability-and-catalogs.md +++ b/docs/source/decisions/0003-searchability-and-catalogs.rst @@ -1,13 +1,18 @@ -# Data should be readily searchable +################################# +Data should be readily searchable +################################# -## Summary +Summary +======= It's not just a good idea, it's technically the law. While CDA currently expose a fair amount of information to search it's never entirely clear. -## Opinions +Opinions +======== -### Opinion 1 +Opinion 1 +--------- Summary: Each data time should support it's own /catalog end point. @@ -22,7 +27,8 @@ document which is for what or what changes for each. To make 'catalog' operations clear, we should create /catalog for each data type that provide for discoverability of that data. -### Opinion 2 +Opinion 2 +--------- Summary: Each datatype under "catalog" should be a full path" @@ -33,10 +39,12 @@ path under `/catalog` instead of the current path parameter is a better approach We would maintain the grouping, but each catalog can have it's appropriate search criteria. -## Decision Status +Decision Status +=============== proposed [comment:] <> (Status: request for comments | proposed | accepted | rejected | deprecated | superseded) -## References +References +========== diff --git a/docs/source/decisions/0004-versioning.rst b/docs/source/decisions/0004-versioning.rst new file mode 100644 index 000000000..55e5defc4 --- /dev/null +++ b/docs/source/decisions/0004-versioning.rst @@ -0,0 +1,61 @@ +######################## +CWMS Data Api Versioning +######################## + + +Summary +======= + +Maintaining backwards compatibilty while improving future difficulty has proven sufficiently difficulty that change +is required. + +The API as a whole will retain the calendar based versioning for formal releases. +Data *SHOULD* be versioned, if appropriate/needed, with otherwise backwards compatible changes to query parameters. +Endpoints will be places under a new "api version" for backwards incompatible or confusing parameter changes. + +e.g. + +`https://host/cwms-data/locations` + +can become + +`https://host/cwms-data/v2/locations` + +As additional endpoints require such a change they should be added to an existing increased version. + +Example: + +given above and a v1 `timeseries` and yet another `locations` improvements + +.. code-block:: bash + + # new time series becomes + cwms-data/v2/timeseries + # the new location becomes + cwms-data/v3/locations + +.. NOTE:: + + Or is that confusing and we should just allows add a new endpoint to the highest endpoint version? + +Opinions +======== + +Opinion 1 +--------- + +Summary: Current scheme is not working + +Author MikeNeilson, on behalf of others + +We have failed to properly handle existing usages while attempting to improve the overall design of the api +and have been breaking various downstream usages due to the confusion. Allowing the endpoints to be versioned allows +an easier time keeping existing behavior while at allowing more drastic improvements in usages to happen. + +Decision Status +=============== + +[comment:] <> (Status: request for comments | proposed | accepted | rejected | deprecated | superseded) + +References +========== diff --git a/docs/decisions/adr-template.md b/docs/source/decisions/adr-template.rst similarity index 53% rename from docs/decisions/adr-template.md rename to docs/source/decisions/adr-template.rst index 6fad01441..1629e8787 100644 --- a/docs/decisions/adr-template.md +++ b/docs/source/decisions/adr-template.rst @@ -1,10 +1,16 @@ -# Title +##### +Title +##### -## Summary -## Opinions +Summary +======= -### Opinion 1 +Opinions +======== + +Opinion 1 +--------- Summary: @@ -12,8 +18,10 @@ Author descriptive text -## Decision Status +Decision Status +=============== [comment:] <> (Status: request for comments | proposed | accepted | rejected | deprecated | superseded) -## References +References +========== diff --git a/docs/source/decisions/index.rst b/docs/source/decisions/index.rst new file mode 100644 index 000000000..647b8df88 --- /dev/null +++ b/docs/source/decisions/index.rst @@ -0,0 +1,23 @@ +################ +Design Decisions +################ + + +Overview +======== + + +Below are agree upon decision choices regarding the usage of the API. +Please note that certain decisions may be agreed upon before implementation. +Whether or not a particular choice is implemented will be marked for each decision record. + +Some decisions may also be a proposal and marked appropriately. + +.. toctree:: + :maxdepth: 1 + :caption: Decisions + + Api Versioning <./0001-api-versioning.rst> + Data Versioning <./0002-data-versioning.rst> + Catalogs and Search <./0003-searchability-and-catalogs.rst> + Versioning <./0004-versioning.rst> diff --git a/docs/source/index.rst b/docs/source/index.rst index c2bfdf963..7afaf859c 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -38,8 +38,14 @@ Welcome to CWMS Data API documentation! .. toctree:: - :maxdepth: 1 - :caption: Alternative Topics + :maxdepth: 1 + :caption: Alternative Topics - Alternative Topics <./alternative-topics/index.rst> + Alternative Topics <./alternative-topics/index.rst> +.. toctree:: + :maxdepth: 1 + :caption: Design + + Design <./introduction/design.rst> + Decision Records <./decisions/index.rst> diff --git a/docs/source/libraries/java.rst b/docs/source/libraries/java.rst index f94f6e318..ee1a88f88 100644 --- a/docs/source/libraries/java.rst +++ b/docs/source/libraries/java.rst @@ -4,5 +4,3 @@ CWMS Java Client Library - cwmsjava ===================================== This page is coming soon. Please check back later for updates and new content. - - From 9ddb27b37cd318613ab386dcb34d9701b1195eed Mon Sep 17 00:00:00 2001 From: Mike Neilson Date: Fri, 5 Sep 2025 15:55:59 +0000 Subject: [PATCH 3/7] Update language and modified documents to reflect reality. --- .../source/decisions/0002-data-versioning.rst | 27 ++++++++++--------- docs/source/decisions/0004-versioning.rst | 2 +- 2 files changed, 16 insertions(+), 13 deletions(-) diff --git a/docs/source/decisions/0002-data-versioning.rst b/docs/source/decisions/0002-data-versioning.rst index d5e28ad4d..4f79849e9 100644 --- a/docs/source/decisions/0002-data-versioning.rst +++ b/docs/source/decisions/0002-data-versioning.rst @@ -5,7 +5,7 @@ Data Types use Calendar Versioning Summary ======= -Instead of versioning the entire API, though the API has a version, we version the data types. +Instead of versioning the entire API, we version the data types if appropriate. Opinions ======== @@ -13,7 +13,8 @@ Opinions Opinion 1 --------- -Summary: Versioning the API itself, at the level of the path, will lead to too many paths to manage and be awkward for the clients +Summary: If a given end point can have additional, or modified, elements in the response, add a data version. + @MikeNeilson @@ -27,29 +28,27 @@ The header, Accept, informs the API what format, or formats, we are willing to a - Decision Status --------------- partial acceptance -Data, by content-types, are versioned. In the past there was some severe confusion on this part and it was treated as anything new was "version=2" in the content-type. To allow this design but reduce confusion going forward +Data, by content-types, are versioned. In the past there was some severe confusion on this part and it was treated as anything +new was "version=2" in the content-type. To allow this design but reduce confusion going forward -1. The initial version of a data set *SHALL* be the date it was finalized (e.g. PR about to be merged.) -2. *IF* it is the first version of this data the plain content-type "e.g. application/json" will point to this data. -3. *IF* is it not the version version of this data, it will be discussed and announced when it becomes the new default data. -4. Downstream systems *SHOULD* use the specific version regardless of when implemented, and this behavior should be well documented. -5. If a given data set includes definitions of its shape within the type there should be sufficient documentation for downstream +1. The initial content-type of a data set *SHALL* be be the plain content-type and *SHOULD* include an additional expanded content-type +3. *IF* is it not the first version of this data, additional information will be set in the content-type as + appropriate to the to the data. (e.g. `application/json+` or `application/json;`) + 1. It will be discussed and announced when it becomes the new default data, if that decision is made. +5. Downstream systems *SHOULD* use the specific version regardless of when implemented, and this behavior should be well documented. +6. If a given data set includes definitions of its shape within the type there should be sufficient documentation for downstream developers to properly account for any changes over time. (See our TimeSeries type and discussions within #927). -Version format is `YYYY-MM-DD` - [comment:] <> (Status: request for comments | proposed | accepted | rejected | deprecated | superseded) References ========== -I have several, I will dig them up likely next week Notes ===== @@ -61,3 +60,7 @@ this was the version for the new system. Attempting to use a simple number of th We also failed to create the initial alias system which caused even more confusion when users attempted to test things directly in a browser instead of the provided swagger-ui. + + +Various practical concerns and common usage have also made doing this "pedantically correct" impossible to manage. The above +should be a reasonable compromise. \ No newline at end of file diff --git a/docs/source/decisions/0004-versioning.rst b/docs/source/decisions/0004-versioning.rst index 55e5defc4..819eee79a 100644 --- a/docs/source/decisions/0004-versioning.rst +++ b/docs/source/decisions/0004-versioning.rst @@ -6,7 +6,7 @@ CWMS Data Api Versioning Summary ======= -Maintaining backwards compatibilty while improving future difficulty has proven sufficiently difficulty that change +Maintaining backwards compatibility while improving future difficulty has proven sufficiently difficulty that change is required. The API as a whole will retain the calendar based versioning for formal releases. From 871a7ce98d8b5f4ac9edf9bf6ccde51305bc16b3 Mon Sep 17 00:00:00 2001 From: Mike Neilson Date: Fri, 7 Nov 2025 06:47:07 -0800 Subject: [PATCH 4/7] Apply suggestions from code review Co-authored-by: Adam Korynta <47677856+adamkorynta@users.noreply.github.com> --- docs/source/decisions/0001-api-versioning.rst | 7 +++++++ docs/source/decisions/0002-data-versioning.rst | 10 ++++++++++ .../decisions/0003-searchability-and-catalogs.rst | 4 ++-- docs/source/decisions/0004-versioning.rst | 4 ++-- docs/source/decisions/index.rst | 2 +- 5 files changed, 22 insertions(+), 5 deletions(-) diff --git a/docs/source/decisions/0001-api-versioning.rst b/docs/source/decisions/0001-api-versioning.rst index 420c5ac74..a0e5a10a6 100644 --- a/docs/source/decisions/0001-api-versioning.rst +++ b/docs/source/decisions/0001-api-versioning.rst @@ -26,6 +26,13 @@ While we can automate SemVer it is an additional step. With Calendar Versioning automation tools can just pick the current date when appropriately triggered, perhaps by merged into a particular branch. +@adamkorynta + + Summary: Calendar versioning more directly aligns with consumers needs + +Deprecation, desupport, and removal notices usually involve timelines rather than predicting x number of major/minor releases into the future. + +Given the endpoints themselves are already versioned, following semver becomes obtuse with only a real use-case of structural changes such as supported JDK changes, etc. Opinion 2 Users --------------- diff --git a/docs/source/decisions/0002-data-versioning.rst b/docs/source/decisions/0002-data-versioning.rst index 4f79849e9..464064998 100644 --- a/docs/source/decisions/0002-data-versioning.rst +++ b/docs/source/decisions/0002-data-versioning.rst @@ -28,6 +28,16 @@ The header, Accept, informs the API what format, or formats, we are willing to a +Opinion 2 + +Summary: Remove data versioning in all endpoints. + +@adamkorynta + +The accept/content-type headers have provided sufficient confusion to downstream clients and deviate from industry standards. I have yet to see an endpoint where this versioning solves the proposed problem of needing to different shapes of data (other than json vs xml). The concept was introduced to solve the straight-to-db queries, which did not have any OpenAPI documentation moving to in-app DTO's which now have documentation. When moving away from the straight-to-db queries, we needed to thoroughly expand queries parameters and make other backwards-incompatible changes unrelated to the data shape. Given that context in hindsight a path version would have worked better. + +The closest we've gotten to needing new shapes on the same endpoint is the data-entry date on TimeSeries, but this was more appropriately solved via query parameter and data arrays. I think if we had really wanted to be a stickler on the "what format" we could have easily added another endpoint path instead. Even if/when we add text annotations, using a content type is obtuse given the lack of discoverability as we would then need `application/json`, `application/json+data-entry+text-annotations`, `application/json+text-annotations`, `application/json+data-entry` which seems like just another type of bloat that is more hidden from clients. + Decision Status --------------- diff --git a/docs/source/decisions/0003-searchability-and-catalogs.rst b/docs/source/decisions/0003-searchability-and-catalogs.rst index 62996ea70..f7ec3a5e3 100644 --- a/docs/source/decisions/0003-searchability-and-catalogs.rst +++ b/docs/source/decisions/0003-searchability-and-catalogs.rst @@ -14,7 +14,7 @@ Opinions Opinion 1 --------- -Summary: Each data time should support it's own /catalog end point. +Summary: Each data type should support it's own /catalog end point. @MikeNeilson @@ -37,7 +37,7 @@ Summary: Each datatype under "catalog" should be a full path" If it makes sense to group all catalogs under catalog, perhaps for grouping in the SWAGGER-UI, making each catalog it's own path under `/catalog` instead of the current path parameter is a better approach. -We would maintain the grouping, but each catalog can have it's appropriate search criteria. +We would maintain the grouping, but each catalog can have its appropriate search criteria. Decision Status =============== diff --git a/docs/source/decisions/0004-versioning.rst b/docs/source/decisions/0004-versioning.rst index 819eee79a..fbd551a23 100644 --- a/docs/source/decisions/0004-versioning.rst +++ b/docs/source/decisions/0004-versioning.rst @@ -11,7 +11,7 @@ is required. The API as a whole will retain the calendar based versioning for formal releases. Data *SHOULD* be versioned, if appropriate/needed, with otherwise backwards compatible changes to query parameters. -Endpoints will be places under a new "api version" for backwards incompatible or confusing parameter changes. +Endpoints will be placed under a new "api version" for backwards incompatible or confusing parameter changes. e.g. @@ -50,7 +50,7 @@ Author MikeNeilson, on behalf of others We have failed to properly handle existing usages while attempting to improve the overall design of the api and have been breaking various downstream usages due to the confusion. Allowing the endpoints to be versioned allows -an easier time keeping existing behavior while at allowing more drastic improvements in usages to happen. +an easier time keeping existing behavior while also allowing more drastic improvements in usages to happen. Decision Status =============== diff --git a/docs/source/decisions/index.rst b/docs/source/decisions/index.rst index 647b8df88..cdf44f85a 100644 --- a/docs/source/decisions/index.rst +++ b/docs/source/decisions/index.rst @@ -7,7 +7,7 @@ Overview ======== -Below are agree upon decision choices regarding the usage of the API. +Below are agreed upon decision choices regarding the usage of the API. Please note that certain decisions may be agreed upon before implementation. Whether or not a particular choice is implemented will be marked for each decision record. From 9085d3b23c72654aac1c372dc71fa36f034ff634 Mon Sep 17 00:00:00 2001 From: Mike Neilson Date: Mon, 17 Nov 2025 16:13:10 +0000 Subject: [PATCH 5/7] Reponses to feedback. --- .../source/decisions/0002-data-versioning.rst | 24 ++++++++--- .../0003-searchability-and-catalogs.rst | 42 ++++++++++++++++--- docs/source/decisions/0004-versioning.rst | 25 ++++++++++- 3 files changed, 78 insertions(+), 13 deletions(-) diff --git a/docs/source/decisions/0002-data-versioning.rst b/docs/source/decisions/0002-data-versioning.rst index 464064998..55e69aafb 100644 --- a/docs/source/decisions/0002-data-versioning.rst +++ b/docs/source/decisions/0002-data-versioning.rst @@ -6,6 +6,8 @@ Summary ======= Instead of versioning the entire API, we version the data types if appropriate. +This versioning only applies to simple non-breaking changes and it primarily presented as a hint in the returned +data. Opinions ======== @@ -26,22 +28,30 @@ e.g /timeseries/Alder Springs.Temp-Air.Inst.15Minutes.0.GOES-raw?begin=PT0&end=P The header, Accept, informs the API what format, or formats, we are willing to accept the data in. - - Opinion 2 +--------- Summary: Remove data versioning in all endpoints. @adamkorynta -The accept/content-type headers have provided sufficient confusion to downstream clients and deviate from industry standards. I have yet to see an endpoint where this versioning solves the proposed problem of needing to different shapes of data (other than json vs xml). The concept was introduced to solve the straight-to-db queries, which did not have any OpenAPI documentation moving to in-app DTO's which now have documentation. When moving away from the straight-to-db queries, we needed to thoroughly expand queries parameters and make other backwards-incompatible changes unrelated to the data shape. Given that context in hindsight a path version would have worked better. +The accept/content-type headers have provided sufficient confusion to downstream clients and deviate from industry standards. +I have yet to see an endpoint where this versioning solves the proposed problem of needing to different shapes of data (other than json vs xml). +The concept was introduced to solve the straight-to-db queries, which did not have any OpenAPI documentation moving to in-app DTO's which now have documentation. When moving away from the straight-to-db queries, we needed to thoroughly expand queries parameters and make other backwards-incompatible changes unrelated to the data shape. Given that context in hindsight a path version would have worked better. -The closest we've gotten to needing new shapes on the same endpoint is the data-entry date on TimeSeries, but this was more appropriately solved via query parameter and data arrays. I think if we had really wanted to be a stickler on the "what format" we could have easily added another endpoint path instead. Even if/when we add text annotations, using a content type is obtuse given the lack of discoverability as we would then need `application/json`, `application/json+data-entry+text-annotations`, `application/json+text-annotations`, `application/json+data-entry` which seems like just another type of bloat that is more hidden from clients. +The closest we've gotten to needing new shapes on the same endpoint is the data-entry date on TimeSeries, but this was +more appropriately solved via query parameter and data arrays. I think if we had really wanted to be a stickler on the +"what format" we could have easily added another endpoint path instead. Even if/when we add text annotations, +using a content type is obtuse given the lack of discoverability as we would then need `application/json`, +`application/json+data-entry+text-annotations`, `application/json+text-annotations`, `application/json+data-entry` +which seems like just another type of bloat that is more hidden from clients. Decision Status --------------- -partial acceptance +`rejected` - the descriptions in this proposal are awkward and it is not clear how to fix them. Additionally as we have +decided to adopt path based versioning and we've made `application/json` or `application/xml` default to the latest desired +the requirement is now moot. Data, by content-types, are versioned. In the past there was some severe confusion on this part and it was treated as anything new was "version=2" in the content-type. To allow this design but reduce confusion going forward @@ -59,6 +69,10 @@ new was "version=2" in the content-type. To allow this design but reduce confusi References ========== +1. https://www.youtube.com/watch?v=jmoxGJ_sLgU +2. https://newsletter.systemdesign.one/p/api-versioning +3. https://www.speakeasy.com/api-design/versioning + Notes ===== diff --git a/docs/source/decisions/0003-searchability-and-catalogs.rst b/docs/source/decisions/0003-searchability-and-catalogs.rst index f7ec3a5e3..03d5041cc 100644 --- a/docs/source/decisions/0003-searchability-and-catalogs.rst +++ b/docs/source/decisions/0003-searchability-and-catalogs.rst @@ -5,8 +5,23 @@ Data should be readily searchable Summary ======= -It's not just a good idea, it's technically the law. While CDA currently expose a fair amount of information to search -it's never entirely clear. +It's not just a good idea, it's technically the law, see reference 1 and 2. While CDA currently expose a fair amount +of information to search it's never entirely clear. We *MUST* adopt a standard method of searchability. + +Additional Information +---------------------- + +Catalog - a complete list of items, for examples of things that people can look at or buy [3] + +By having a well defined structure of information users can more easily discover what they are looking for. While the +Sagger-UI, if used, presented all of the types of data that can be found. The `catalog` for each type should present +a clear way to find the available data of each type. + +The catalog of each data set would include only metadata associated with each data type. For example a time series +catalog would include support to discover primary timeseries names, aliases, extends, and the like but not actual time +series data. + +However, for data that are primarily metadata, such as Locations, they would be one and the same. Opinions ======== @@ -19,7 +34,7 @@ Summary: Each data type should support it's own /catalog end point. @MikeNeilson The original CDA has the say, `/timeseries` end point provide a catalog if no data is set. I created a /catalog end point -to attempt to consildate search query parameters. For TimeSeries and Locations this works reasonably well since there +to attempt to consolidate search query parameters. For TimeSeries and Locations this works reasonably well since there is parity between the concepts. However, if we tried to add ratings into the mix, the list of query parameters grows, and it would rather difficult to @@ -30,21 +45,36 @@ To make 'catalog' operations clear, we should create /catalog for each data type Opinion 2 --------- -Summary: Each datatype under "catalog" should be a full path" +Summary: Each datatype should exit under a "/catalog" @MikeNeilson If it makes sense to group all catalogs under catalog, perhaps for grouping in the SWAGGER-UI, making each catalog it's own path under `/catalog` instead of the current path parameter is a better approach. -We would maintain the grouping, but each catalog can have its appropriate search criteria. +We would maintain the grouping, but each catalog can have its appropriate search criteria. the `catalog/` could +just redirect. + +Opinion 3 +--------- + +Summary: Swagger-UI allows grouping under multiple tasks + +@Mike Neilson on behalf of @adamkorynta + +We can group the available catalogs into multiple Swagger-UI blocks while maintaining a `/catalog` Decision Status =============== -proposed +proposed - requires additional discussion and likely some review after the first path based version is adopted. +Document is left in proposed state to indicate additional ideas should be presented over time and as work is done. [comment:] <> (Status: request for comments | proposed | accepted | rejected | deprecated | superseded) References ========== + +1. https://www.congress.gov/bill/115th-congress/house-bill/1770 +2. https://www.cio.gov/handbook/it-laws/ogda/ +3. https://www.oxfordlearnersdictionaries.com/us/definition/english/catalogue_1 \ No newline at end of file diff --git a/docs/source/decisions/0004-versioning.rst b/docs/source/decisions/0004-versioning.rst index fbd551a23..091f67723 100644 --- a/docs/source/decisions/0004-versioning.rst +++ b/docs/source/decisions/0004-versioning.rst @@ -11,7 +11,7 @@ is required. The API as a whole will retain the calendar based versioning for formal releases. Data *SHOULD* be versioned, if appropriate/needed, with otherwise backwards compatible changes to query parameters. -Endpoints will be placed under a new "api version" for backwards incompatible or confusing parameter changes. +Endpoints will be placed under a new "api version" path parameter for backwards incompatible or confusing parameter changes. e.g. @@ -21,7 +21,8 @@ can become `https://host/cwms-data/v2/locations` -As additional endpoints require such a change they should be added to an existing increased version. +As additional endpoints require such a change they should be added to an existing increased version. For each +version all required verbs *SHALL* be implemented. e.g. the new version is a complete unit of operation. Example: @@ -38,6 +39,22 @@ given above and a v1 `timeseries` and yet another `locations` improvements Or is that confusing and we should just allows add a new endpoint to the highest endpoint version? +At Current time the "root" URL will be considered V1, and redirect to v1 urls. +After X years the root URLs will redirect to the latest version. + +e.g. + + .. code-block:: bash + # now + curl "https://cwms-data.usace.army/cwms-data/timeseries/Black Butte.Stor.Inst.~1Day.0.Calc-val?units=ft" + # will redirect to + curl "https://cwms-data.usace.army/cwms-data/v1/timeseries/Black Butte.Stor.Inst.~1Day.0.Calc-val?units=ft" + # after transition period, *IF* there is a new version + # will redirect to + curl "https://cwms-data.usace.army/cwms-data/v/timeseries/Black Butte.Stor.Inst.~1Day.0.Calc-val?units=ft" + # if possible, query parameters can be updated on behalf of the user + + Opinions ======== @@ -59,3 +76,7 @@ Decision Status References ========== + +1. https://www.youtube.com/watch?v=jmoxGJ_sLgU +2. https://newsletter.systemdesign.one/p/api-versioning +3. https://www.speakeasy.com/api-design/versioning From 5aca31e0a89579d9f50e18b2516e77e6887e1e02 Mon Sep 17 00:00:00 2001 From: Mike Neilson Date: Wed, 3 Dec 2025 17:56:47 +0000 Subject: [PATCH 6/7] File correction. --- docs/source/data/index.rst | 17 +---------------- 1 file changed, 1 insertion(+), 16 deletions(-) diff --git a/docs/source/data/index.rst b/docs/source/data/index.rst index a01531ddc..21640bd2a 100644 --- a/docs/source/data/index.rst +++ b/docs/source/data/index.rst @@ -1,6 +1,5 @@ -<<<<<<< HEAD Data Overview -================ +============= .. toctree:: :maxdepth: 1 @@ -8,17 +7,3 @@ Data Overview Timeseries <./timeseries.rst> Location Levels <./location-levels.rst> -======= -#### -Data -#### - - -.. toctree: - :maxdepth: 1 - :caption: Data - - - Location Levels <./location-levels.rst> - Time Series <./timeseries.rst> ->>>>>>> 2fcd1efd (Update decision records to rst.) From 347de19d40ca4e818bb9cfec4a5a434bc9576b06 Mon Sep 17 00:00:00 2001 From: Mike Neilson Date: Wed, 3 Dec 2025 18:03:36 +0000 Subject: [PATCH 7/7] Various updates. --- docs/source/decisions/0003-searchability-and-catalogs.rst | 4 ++-- docs/source/decisions/0004-versioning.rst | 2 ++ docs/source/decisions/index.rst | 2 +- 3 files changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/source/decisions/0003-searchability-and-catalogs.rst b/docs/source/decisions/0003-searchability-and-catalogs.rst index 03d5041cc..679f637c1 100644 --- a/docs/source/decisions/0003-searchability-and-catalogs.rst +++ b/docs/source/decisions/0003-searchability-and-catalogs.rst @@ -14,11 +14,11 @@ Additional Information Catalog - a complete list of items, for examples of things that people can look at or buy [3] By having a well defined structure of information users can more easily discover what they are looking for. While the -Sagger-UI, if used, presented all of the types of data that can be found. The `catalog` for each type should present +Swagger-UI, if used, presented all of the types of data that can be found. The `catalog` for each type should present a clear way to find the available data of each type. The catalog of each data set would include only metadata associated with each data type. For example a time series -catalog would include support to discover primary timeseries names, aliases, extends, and the like but not actual time +catalog would include support to discover primary timeseries names, aliases, extents, and the like but not actual time series data. However, for data that are primarily metadata, such as Locations, they would be one and the same. diff --git a/docs/source/decisions/0004-versioning.rst b/docs/source/decisions/0004-versioning.rst index 091f67723..a443c367f 100644 --- a/docs/source/decisions/0004-versioning.rst +++ b/docs/source/decisions/0004-versioning.rst @@ -72,6 +72,8 @@ an easier time keeping existing behavior while also allowing more drastic improv Decision Status =============== +Status: accepted + [comment:] <> (Status: request for comments | proposed | accepted | rejected | deprecated | superseded) References diff --git a/docs/source/decisions/index.rst b/docs/source/decisions/index.rst index cdf44f85a..80a24751a 100644 --- a/docs/source/decisions/index.rst +++ b/docs/source/decisions/index.rst @@ -18,6 +18,6 @@ Some decisions may also be a proposal and marked appropriately. :caption: Decisions Api Versioning <./0001-api-versioning.rst> - Data Versioning <./0002-data-versioning.rst> + Data Versioning <./0002-data-versioning.rst> (rejected, remains for historical context.) Catalogs and Search <./0003-searchability-and-catalogs.rst> Versioning <./0004-versioning.rst>