diff --git a/docs/source/data/index.rst b/docs/source/data/index.rst index f8a53e2f0..21640bd2a 100644 --- a/docs/source/data/index.rst +++ b/docs/source/data/index.rst @@ -1,5 +1,5 @@ Data Overview -================ +============= .. toctree:: :maxdepth: 1 diff --git a/docs/source/decisions/0001-api-versioning.rst b/docs/source/decisions/0001-api-versioning.rst new file mode 100644 index 000000000..a0e5a10a6 --- /dev/null +++ b/docs/source/decisions/0001-api-versioning.rst @@ -0,0 +1,60 @@ +############################ +API Versioning and Reporting +############################ + +Summary +======= + +API should provide a reason version and list/matrix of capabilities for a given instance of CDA + +Opinions +======== + +Opinion 1 Calendar versioning +----------------------------- + +@MikeNeilson + +Summary: Calendar versioning is easier to support and automate with this API + +As the one whose has been making the "official" releases Semantic Version versioning has basically been useless. +We have been making so many feature additions that if I was doing it right we'd never have a minor version change between releases. +It's also caused me to, I think, release too slowly. + +While we can automate SemVer it is an additional step. + +With Calendar Versioning automation tools can just pick the current date when appropriately triggered, +perhaps by merged into a particular branch. + +@adamkorynta + + Summary: Calendar versioning more directly aligns with consumers needs + +Deprecation, desupport, and removal notices usually involve timelines rather than predicting x number of major/minor releases into the future. + +Given the endpoints themselves are already versioned, following semver becomes obtuse with only a real use-case of structural changes such as supported JDK changes, etc. + +Opinion 2 Users +--------------- + +Summary: It has been asked more than once that a version be provided + +Having a version allows client to better respond to what's available instead of failing in obtuse ways. + + +Decision Status +=============== + +accepted + +1. Provide endpoint to retrieve current API version. +2. Likely include capability list or matrix. + +References +========== + + +Related decisions +================= + +- data-versioning diff --git a/docs/source/decisions/0002-data-versioning.rst b/docs/source/decisions/0002-data-versioning.rst new file mode 100644 index 000000000..55e69aafb --- /dev/null +++ b/docs/source/decisions/0002-data-versioning.rst @@ -0,0 +1,90 @@ +################################## +Data Types use Calendar Versioning +################################## + +Summary +======= + +Instead of versioning the entire API, we version the data types if appropriate. +This versioning only applies to simple non-breaking changes and it primarily presented as a hint in the returned +data. + +Opinions +======== + +Opinion 1 +--------- + +Summary: If a given end point can have additional, or modified, elements in the response, add a data version. + + +@MikeNeilson + +By versioning the data, and using the Content-Type and Accept headers and the full features of MIME types we appropriately +separate the concern of "what data we are retrieving/storing" from the presentation of data. + +e.g /timeseries/Alder Springs.Temp-Air.Inst.15Minutes.0.GOES-raw?begin=PT0&end=PT-1D&units=C, granted with a reasonable + exception to the units, defines *what* we want. + +The header, Accept, informs the API what format, or formats, we are willing to accept the data in. + +Opinion 2 +--------- + +Summary: Remove data versioning in all endpoints. + +@adamkorynta + +The accept/content-type headers have provided sufficient confusion to downstream clients and deviate from industry standards. +I have yet to see an endpoint where this versioning solves the proposed problem of needing to different shapes of data (other than json vs xml). +The concept was introduced to solve the straight-to-db queries, which did not have any OpenAPI documentation moving to in-app DTO's which now have documentation. When moving away from the straight-to-db queries, we needed to thoroughly expand queries parameters and make other backwards-incompatible changes unrelated to the data shape. Given that context in hindsight a path version would have worked better. + +The closest we've gotten to needing new shapes on the same endpoint is the data-entry date on TimeSeries, but this was +more appropriately solved via query parameter and data arrays. I think if we had really wanted to be a stickler on the +"what format" we could have easily added another endpoint path instead. Even if/when we add text annotations, +using a content type is obtuse given the lack of discoverability as we would then need `application/json`, +`application/json+data-entry+text-annotations`, `application/json+text-annotations`, `application/json+data-entry` +which seems like just another type of bloat that is more hidden from clients. + +Decision Status +--------------- + +`rejected` - the descriptions in this proposal are awkward and it is not clear how to fix them. Additionally as we have +decided to adopt path based versioning and we've made `application/json` or `application/xml` default to the latest desired +the requirement is now moot. + +Data, by content-types, are versioned. In the past there was some severe confusion on this part and it was treated as anything +new was "version=2" in the content-type. To allow this design but reduce confusion going forward + +1. The initial content-type of a data set *SHALL* be be the plain content-type and *SHOULD* include an additional expanded content-type +3. *IF* is it not the first version of this data, additional information will be set in the content-type as + appropriate to the to the data. (e.g. `application/json+` or `application/json;`) + 1. It will be discussed and announced when it becomes the new default data, if that decision is made. +5. Downstream systems *SHOULD* use the specific version regardless of when implemented, and this behavior should be well documented. +6. If a given data set includes definitions of its shape within the type there should be sufficient documentation for downstream + developers to properly account for any changes over time. (See our TimeSeries type and discussions within #927). + +[comment:] <> (Status: request for comments | proposed | accepted | rejected | deprecated | superseded) + +References +========== + +1. https://www.youtube.com/watch?v=jmoxGJ_sLgU +2. https://newsletter.systemdesign.one/p/api-versioning +3. https://www.speakeasy.com/api-design/versioning + + +Notes +===== + +The initial idea in CDA was that the first version of any data type was, we'll just stick with JSON for each of message, +"application/json;version=1" with "application/json" being the alias to the latest format version. However, this was not +correctly communicated and several brand new data transfer objects were created as ";version=2" under the impression that +this was the version for the new system. Attempting to use a simple number of this has clearly caused confusion in general. + +We also failed to create the initial alias system which caused even more confusion when users attempted to test things +directly in a browser instead of the provided swagger-ui. + + +Various practical concerns and common usage have also made doing this "pedantically correct" impossible to manage. The above +should be a reasonable compromise. \ No newline at end of file diff --git a/docs/source/decisions/0003-searchability-and-catalogs.rst b/docs/source/decisions/0003-searchability-and-catalogs.rst new file mode 100644 index 000000000..679f637c1 --- /dev/null +++ b/docs/source/decisions/0003-searchability-and-catalogs.rst @@ -0,0 +1,80 @@ +################################# +Data should be readily searchable +################################# + +Summary +======= + +It's not just a good idea, it's technically the law, see reference 1 and 2. While CDA currently expose a fair amount +of information to search it's never entirely clear. We *MUST* adopt a standard method of searchability. + +Additional Information +---------------------- + +Catalog - a complete list of items, for examples of things that people can look at or buy [3] + +By having a well defined structure of information users can more easily discover what they are looking for. While the +Swagger-UI, if used, presented all of the types of data that can be found. The `catalog` for each type should present +a clear way to find the available data of each type. + +The catalog of each data set would include only metadata associated with each data type. For example a time series +catalog would include support to discover primary timeseries names, aliases, extents, and the like but not actual time +series data. + +However, for data that are primarily metadata, such as Locations, they would be one and the same. + +Opinions +======== + +Opinion 1 +--------- + +Summary: Each data type should support it's own /catalog end point. + +@MikeNeilson + +The original CDA has the say, `/timeseries` end point provide a catalog if no data is set. I created a /catalog end point +to attempt to consolidate search query parameters. For TimeSeries and Locations this works reasonably well since there +is parity between the concepts. + +However, if we tried to add ratings into the mix, the list of query parameters grows, and it would rather difficult to +document which is for what or what changes for each. + +To make 'catalog' operations clear, we should create /catalog for each data type that provide for discoverability of that data. + +Opinion 2 +--------- + +Summary: Each datatype should exit under a "/catalog" + +@MikeNeilson + +If it makes sense to group all catalogs under catalog, perhaps for grouping in the SWAGGER-UI, making each catalog it's own +path under `/catalog` instead of the current path parameter is a better approach. + +We would maintain the grouping, but each catalog can have its appropriate search criteria. the `catalog/` could +just redirect. + +Opinion 3 +--------- + +Summary: Swagger-UI allows grouping under multiple tasks + +@Mike Neilson on behalf of @adamkorynta + +We can group the available catalogs into multiple Swagger-UI blocks while maintaining a `/catalog` + +Decision Status +=============== + +proposed - requires additional discussion and likely some review after the first path based version is adopted. +Document is left in proposed state to indicate additional ideas should be presented over time and as work is done. + +[comment:] <> (Status: request for comments | proposed | accepted | rejected | deprecated | superseded) + +References +========== + +1. https://www.congress.gov/bill/115th-congress/house-bill/1770 +2. https://www.cio.gov/handbook/it-laws/ogda/ +3. https://www.oxfordlearnersdictionaries.com/us/definition/english/catalogue_1 \ No newline at end of file diff --git a/docs/source/decisions/0004-versioning.rst b/docs/source/decisions/0004-versioning.rst new file mode 100644 index 000000000..a443c367f --- /dev/null +++ b/docs/source/decisions/0004-versioning.rst @@ -0,0 +1,84 @@ +######################## +CWMS Data Api Versioning +######################## + + +Summary +======= + +Maintaining backwards compatibility while improving future difficulty has proven sufficiently difficulty that change +is required. + +The API as a whole will retain the calendar based versioning for formal releases. +Data *SHOULD* be versioned, if appropriate/needed, with otherwise backwards compatible changes to query parameters. +Endpoints will be placed under a new "api version" path parameter for backwards incompatible or confusing parameter changes. + +e.g. + +`https://host/cwms-data/locations` + +can become + +`https://host/cwms-data/v2/locations` + +As additional endpoints require such a change they should be added to an existing increased version. For each +version all required verbs *SHALL* be implemented. e.g. the new version is a complete unit of operation. + +Example: + +given above and a v1 `timeseries` and yet another `locations` improvements + +.. code-block:: bash + + # new time series becomes + cwms-data/v2/timeseries + # the new location becomes + cwms-data/v3/locations + +.. NOTE:: + + Or is that confusing and we should just allows add a new endpoint to the highest endpoint version? + +At Current time the "root" URL will be considered V1, and redirect to v1 urls. +After X years the root URLs will redirect to the latest version. + +e.g. + + .. code-block:: bash + # now + curl "https://cwms-data.usace.army/cwms-data/timeseries/Black Butte.Stor.Inst.~1Day.0.Calc-val?units=ft" + # will redirect to + curl "https://cwms-data.usace.army/cwms-data/v1/timeseries/Black Butte.Stor.Inst.~1Day.0.Calc-val?units=ft" + # after transition period, *IF* there is a new version + # will redirect to + curl "https://cwms-data.usace.army/cwms-data/v/timeseries/Black Butte.Stor.Inst.~1Day.0.Calc-val?units=ft" + # if possible, query parameters can be updated on behalf of the user + + +Opinions +======== + +Opinion 1 +--------- + +Summary: Current scheme is not working + +Author MikeNeilson, on behalf of others + +We have failed to properly handle existing usages while attempting to improve the overall design of the api +and have been breaking various downstream usages due to the confusion. Allowing the endpoints to be versioned allows +an easier time keeping existing behavior while also allowing more drastic improvements in usages to happen. + +Decision Status +=============== + +Status: accepted + +[comment:] <> (Status: request for comments | proposed | accepted | rejected | deprecated | superseded) + +References +========== + +1. https://www.youtube.com/watch?v=jmoxGJ_sLgU +2. https://newsletter.systemdesign.one/p/api-versioning +3. https://www.speakeasy.com/api-design/versioning diff --git a/docs/source/decisions/adr-template.rst b/docs/source/decisions/adr-template.rst new file mode 100644 index 000000000..1629e8787 --- /dev/null +++ b/docs/source/decisions/adr-template.rst @@ -0,0 +1,27 @@ +##### +Title +##### + + +Summary +======= + +Opinions +======== + +Opinion 1 +--------- + +Summary: + +Author + +descriptive text + +Decision Status +=============== + +[comment:] <> (Status: request for comments | proposed | accepted | rejected | deprecated | superseded) + +References +========== diff --git a/docs/source/decisions/index.rst b/docs/source/decisions/index.rst new file mode 100644 index 000000000..80a24751a --- /dev/null +++ b/docs/source/decisions/index.rst @@ -0,0 +1,23 @@ +################ +Design Decisions +################ + + +Overview +======== + + +Below are agreed upon decision choices regarding the usage of the API. +Please note that certain decisions may be agreed upon before implementation. +Whether or not a particular choice is implemented will be marked for each decision record. + +Some decisions may also be a proposal and marked appropriately. + +.. toctree:: + :maxdepth: 1 + :caption: Decisions + + Api Versioning <./0001-api-versioning.rst> + Data Versioning <./0002-data-versioning.rst> (rejected, remains for historical context.) + Catalogs and Search <./0003-searchability-and-catalogs.rst> + Versioning <./0004-versioning.rst> diff --git a/docs/source/index.rst b/docs/source/index.rst index c2bfdf963..7afaf859c 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -38,8 +38,14 @@ Welcome to CWMS Data API documentation! .. toctree:: - :maxdepth: 1 - :caption: Alternative Topics + :maxdepth: 1 + :caption: Alternative Topics - Alternative Topics <./alternative-topics/index.rst> + Alternative Topics <./alternative-topics/index.rst> +.. toctree:: + :maxdepth: 1 + :caption: Design + + Design <./introduction/design.rst> + Decision Records <./decisions/index.rst> diff --git a/docs/source/libraries/java.rst b/docs/source/libraries/java.rst index f94f6e318..ee1a88f88 100644 --- a/docs/source/libraries/java.rst +++ b/docs/source/libraries/java.rst @@ -4,5 +4,3 @@ CWMS Java Client Library - cwmsjava ===================================== This page is coming soon. Please check back later for updates and new content. - -