Output of `optic diff` looks like a successful run, but it hides the errors that causes exit code 1

[gabsong]

Describe the bug
I think this is a bug, but it may be open to interpretation. Running:

optic diff --base <branch> <file_to_compare_against> --check

Gave the following output:

x Accounts API openapi/federation/external/swagger.2023-01.json
Preview docs:  https://useoptic.com/docs/cloud-get-started
Operations: 1 operation added
x  Checks: 36/48 passed (11 warnings)

specification details:
- /servers, /info/contact added
- /openapi, /info/version, /info/title changed

✔ POST /logistics/api/2023-01/accounts: added

  - request:
    - body application/json:
  - response 200:
    - body application/json:
  - response 401:
    - body application/json:
  - response 409:
    - body application/json:

...and 1 other endpoint

See the full history of this API by running "optic api add openapi/federation/external/swagger.2023-01.json --history-depth 0"

If I check the exit code, I get 1.

$ echo $?
1

To Reproduce
N/A

Expected behavior
I thought if there is an error causing the exit code 1, getting the output would be helpful. Or the exit code may be set on warnings? Because I do have warnings when running optic lint.

Screenshots

Details (please complete the following information):

• OS and version: Ubuntu (CircleCI virtual env)
• Optic version: 0.47.3
• NodeJS version: 18

Additional context
For context, there's no file to compare to in the staging branch. Also, ran optic lint to check for errors, but there were none. I think the exit code may be wrong, or the output is hiding the error.

[niclim]

Hi - it looks like the output isn't showing the rule errors - let me look into this.

If its possible, could you provide:

• the OpenAPI spec
• the ruleset you are using to run this (this is usually in your optic.yml file)

That'd be helpful to confirm should be rendered here

[gabsong]

Hi, I have more information and a clearer hypothesis for why this happened. It's a different run (meaning the spec has changed since), but I was able to obtain more information.

Here's the cli output:

$ npx optic diff ./openapi/federation/external/swagger.unstable.json --check --web

x Logistics API ./openapi/federation/external/swagger.unstable.json
Operations: 10 operations added
x  Checks: 192/297 passed (96 warnings)

specification details:
- /servers, /info/contact added
- /openapi, /info/version, /info/title changed

✔ POST /logistics/api/unstable/accounts: added

  - request:
    - body application/json:
  - response 200:
    - body application/json:
  - response 401:
    - body application/json:
  - response 409:
    - body application/json:

✔ GET /logistics/api/unstable/accounts/by_seller_id/{sellerId}: added

  - response 200:
    - body application/json:
  - response 401:
    - body application/json:
  - response 404:
    - body application/json:

✔ GET /logistics/api/unstable/accounts/by_seller_id/{sellerId}/is_seller_portal_account: added

  - response 200:
    - body application/json:
  - response 401:
    - body application/json:
  - response 404:
    - body application/json:

✔ GET /logistics/api/unstable/accounts/unified_seller_ids: added

  - query parameter startCreatedAt:
  - response 401:
    - body application/json:

✔ GET /logistics/api/unstable/accounts/unified_billing: added

  - query parameter startCreatedAt:
  - response 200:
    - body application/json:
  - response 401:
    - body application/json:

✔ GET /logistics/api/unstable/accounts/{logisticsAccountId}: added

  - response 200:
    - body application/json:
  - response 401:
    - body application/json:
  - response 404:
    - body application/json:

✔ GET /logistics/api/unstable/accounts/{logisticsAccountId}/seller_id: added

  - response 200:
    - body application/json:
  - response 401:
    - body application/json:
  - response 404:
    - body application/json:

✔ PUT /logistics/api/unstable/accounts/{logisticsAccountId}/is_test_account: added

  - request:
    - body application/json:
  - response 401:
    - body application/json:
  - response 404:
    - body application/json:

✔ PUT /logistics/api/unstable/accounts/{logisticsAccountId}/is_seller_portal_account: added

  - request:
    - body application/json:
  - response 401:
    - body application/json:
  - response 404:
    - body application/json:

✔ PUT /logistics/api/unstable/accounts/{logisticsAccountId}/status: added

  - request:
    - body application/json:
  - response 401:
    - body application/json:
  - response 404:
    - body application/json:

...and 10 other endpoints

See the full history of this API by running "optic api add openapi/federation/external/swagger.unstable.json --history-depth 0"
Opening up diff in web view

gsong@mbp in ~/project (staging●)
$ echo $?
1

Here's the web output.

The web has more information that was not visible on the cli, but since this runs in CI we did not know what was going on and thought it was failing silently.

Looks like execution is exiting with an error due to violations of rules in spectral "added.yml". I can see the red errors in the web, which were not shown in the terminal.

Will change the issue title to be more descriptive of the issue.

[gabsong]

Unrelated question, but is it possible to also output the web URL in the cli when using the option --web?

[acunniffe]

Thanks @gabsong - the web output should give us what we need to debug this properly. We'll get the CLI output fixed.

Re:

is it possible to also output the web URL in the cli when using the option --web

Currently we don't upload any data to Optic's servers when you use --web. The data is kept in the URL fragment which is good for privacy, but bad for portability because the URLs end up so long.

If you want to see the changelog + docs links in CI, you can use the --upload flag (see example). That prints shorter URLs to the CI logs, and (optionally) comments them to the PR feed. Is that something you're interested in?

[gabsong]

If you want to see the changelog + docs links in CI, you can use the --upload flag (see example). That prints shorter URLs to the CI logs, and (optionally) comments them to the PR feed. Is that something you're interested in?

Thanks Aidan! I can look at the --upload flag. Will probably know more after I try it! Is the example link correct? I get another perhaps unrelated PR.

[acunniffe]

Great. Let us know if you need help setting it up.

It requires a token from Optic Cloud. Explained here: https://www.useoptic.com/docs/cloud-get-started#3-turn-on-cloud

The example I sent was just to show you what it looks like after you upload. It's from an example repo we use during demos.

[gabsong]

I see, will take a look :)

[niclim]

Hi @gabsong - I released a new version 0.47.16 that I think should fix the issue of missing errors - could you take a look and confirm whether the new version fixes the issue you are seeing?

[niclim]

Going to close this out first since I believe this fixes the issue, but if this doesn't fix the problem please feel free to reopen the issue - thanks!