Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .dotstop_extensions/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -232,7 +232,7 @@ The expected configuration is:
references:
- type: workflow_failures
owner: "eclipse-score"
repo: "inc_nlohmann_json"
repo: "nlohmann_json"
branch: "json_version_3_12_0"
---
```
Expand Down Expand Up @@ -368,7 +368,7 @@ evidence:
type: did_workflows_fail
configuration:
owner: "eclipse-score" # owner of the repository
repo: "inc_nlohmann_json" # name of the repository
repo: "nlohmann_json" # name of the repository
branch: "json_version_3_12_0" # name of the branch
action: "push" # optional, default is push
```
Expand Down
8 changes: 4 additions & 4 deletions TSF/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -165,7 +165,7 @@ For the error-free execution is it necessary, however, to adhere to the naming s

Based on the above observations, the following steps are recommended for each update to the library.

1. Merge branch master from the original nlohmann/json into an external fork of the eclipse-score/inc_nlohmann_json repository, where steps 2-12 shall be performed.
1. Merge branch master from the original nlohmann/json into an external fork of the eclipse-score/nlohmann_json repository, where steps 2-12 shall be performed.
2. Confirm the deletion of cifuzz.yml, macos.yml and windows.yml.
3. Resolve the potential merge conflict in publish-documentation.yml by rejecting the incoming changes.
4. Update the versions of the github actions, if necessary.
Expand All @@ -178,8 +178,8 @@ Based on the above observations, the following steps are recommended for each up
11. Generate the TSF report locally and carefully investigate any change in the trustable score(s).
If any relevant behaviour of the library changes, adapt the documentation.
Additionally, if any additional tests were added, or existing tests were changed, carefully investigate whether these warrant an amendment of the documentation.
12. Merge into the ``main`` branch of eclipse-score/inc_nlohmann_json.
13. Create a new release tag in eclipse-score/inc_nlohmann_json, following semantic versioning.
12. Merge into the ``main`` branch of eclipse-score/nlohmann_json.
13. Create a new release tag in eclipse-score/nlohmann_json, following semantic versioning.

# Update concept for the TSF documentation

Expand All @@ -192,7 +192,7 @@ Furthermore, the automatic generation of the TSF report and the tracking of chan
## Version of trudag

The documentation is currently built using trudag version v2025.10.22
In case of new releases of trudag in the future, it is recommended to carefully review the introduced changes and rigorously test it before merging it into eclipse-score/inc_nlohmann_json.
In case of new releases of trudag in the future, it is recommended to carefully review the introduced changes and rigorously test it before merging it into eclipse-score/nlohmann_json.
The following should be considered:

* How has the algorithm for the accumulation of the trustable score changed? Ideally, it is not changed, otherwise the necessity for a new review arises.
Expand Down
8 changes: 4 additions & 4 deletions TSF/docs/ci_failure_rate_analysis.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,11 +60,11 @@ In this period, `ci_test_clang` ran 25 times and failed 3 times (12%). All failu
For the selected three-month window before the v3.12.0 release, the Ubuntu CI for `nlohmann/json` shows a small set of jobs with noticeably higher failure rates, but in all cases these can be traced back to intentional CI/tooling work or strict analysis settings, not to undetected misbehaviour of the library itself. All of these high failure rates occurred on PR branches, were visible to developers, and were resolved before merging, while the majority of other jobs (including `ci_test_gcc`, `ci_test_standards_*` and most compiler-matrix entries) remain at or near 0 %. Taken together, this indicates a stable CI system that reacts as intended to real issues and configuration changes, with no evidence of systematic, unexplained spikes in test failures for the Ubuntu workflow.


# Failure rate analysis for eclipse-score/inc_nlohmann_json Parent-Workflow/Ubuntu CI
# Failure rate analysis for eclipse-score/nlohmann_json Parent-Workflow/Ubuntu CI

## Scope and data source

- **Repository:** `eclipse-score/inc_nlohmann_json`
- **Repository:** `eclipse-score/nlohmann_json`
- **Workflow:** `Parent Workflow` (top-level CI workflow)
- **Date range:** last 90 days from 08.12.2025
- **Filter:** `workflow_file_name: parent-workflow.yml`
Expand All @@ -84,7 +84,7 @@ From the Jobs table (sorted by Failure rate):
| `Run Labeler Workflow / clone_missing_labels` | **3%** | ≈ 6 s | 38 |

All other Parent-Workflow jobs in the last 90 days report a failure rate of
**0 %** over 38 runs each. The remaining data can be found on the GitHub [insights page](https://github.com/eclipse-score/inc_nlohmann_json/actions/metrics/performance?dateRangeType=DATE_RANGE_TYPE_LAST_90_DAYS&sort=failureRate&tab=jobs&filters=workflow_file_name%3Aparent-workflow.yml).
**0 %** over 38 runs each. The remaining data can be found on the GitHub [insights page](https://github.com/eclipse-score/nlohmann_json/actions/metrics/performance?dateRangeType=DATE_RANGE_TYPE_LAST_90_DAYS&sort=failureRate&tab=jobs&filters=workflow_file_name%3Aparent-workflow.yml).

## Interpretation of the higher failure rates

Expand All @@ -108,6 +108,6 @@ Taken together, a period of there months show that all test-related
jobs in the Parent Workflow have 0 % failures, while the few non-zero
failure rates are confined to meta-jobs that handle publishing of historical
test data and label synchronization. This indicates a stable CI setup for
`inc_nlohmann_json`, with the only reported failures occurring in
`nlohmann_json`, with the only reported failures occurring in
infrastructure-side integration steps rather than in the core test pipeline.

2 changes: 1 addition & 1 deletion TSF/scripts/test_scripts.py
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ def ET_Element_test():
def ET_Element_nontest():
content = """
<testcase name="download_test_data" classname="download_test_data" time="9.25104" status="run">
<system-out>[1/1] Downloading test data from https://github.com/eclipse-score/inc_nlohmann_json (branch: json_test_data_version_3_1_0_mirror)
<system-out>[1/1] Downloading test data from https://github.com/eclipse-score/nlohmann_json (branch: json_test_data_version_3_1_0_mirror)
</system-out>
</testcase>
"""
Expand Down
2 changes: 1 addition & 1 deletion TSF/trustable/assertions/TA-ANALYSIS.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,4 @@ level: 1.1
normative: true
---

Collected data from tests and monitoring of deployed software in eclipse-score/inc_nlohmann_json is analysed according to specified objectives.
Collected data from tests and monitoring of deployed software in eclipse-score/nlohmann_json is analysed according to specified objectives.
2 changes: 1 addition & 1 deletion TSF/trustable/assertions/TA-ANALYSIS_CONTEXT.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ by confirming that test results remain unchanged when no changes are intended.
- Analysis of failures
- **Answer**: Provided by JLS-26 and JLS-17.
- Analysis of spikes and trends
- **Answer**: CI test failure rates for the upstream `nlohmann/json` repository and `eclipse-score/inc_nlohmann_json` are analysed using the time-series based GitHub Actions metrics views. This analysis is performed manually (see JLS-17). There is currently no fully automated, continuous analysis of failures.
- **Answer**: CI test failure rates for the upstream `nlohmann/json` repository and `eclipse-score/nlohmann_json` are analysed using the time-series based GitHub Actions metrics views. This analysis is performed manually (see JLS-17). There is currently no fully automated, continuous analysis of failures.
- Validation of analysis methods used
- **Answer**: There is no separate formal validation of the analysis methods.

Expand Down
2 changes: 1 addition & 1 deletion TSF/trustable/assertions/TA-CONSTRAINTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,4 @@ level: 1.1
normative: true
---

Constraints on adaptation and deployment of eclipse-score/inc_nlohmann_json are specified.
Constraints on adaptation and deployment of eclipse-score/nlohmann_json are specified.
2 changes: 1 addition & 1 deletion TSF/trustable/assertions/TA-DATA.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,4 @@ level: 1.1
normative: true
---

Data in eclipse-score/inc_nlohmann_json is collected from tests, and from monitoring of deployed software, according to specified objectives.
Data in eclipse-score/nlohmann_json is collected from tests, and from monitoring of deployed software, according to specified objectives.
2 changes: 1 addition & 1 deletion TSF/trustable/assertions/TA-INDICATORS.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,4 @@ level: 1.1
normative: true
---

In eclipse-score/inc_nlohmann_json, advanced warning indicators for misbehaviours are identified, and monitoring mechanisms are specified, verified and validated based on analysis.
In eclipse-score/nlohmann_json, advanced warning indicators for misbehaviours are identified, and monitoring mechanisms are specified, verified and validated based on analysis.
2 changes: 1 addition & 1 deletion TSF/trustable/assertions/TA-ITERATIONS_CONTEXT.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,5 +74,5 @@ Confidence scoring for TA-ITERATIONS based on
- **Answer**: The nlohmann/json library has no external dependencies/accesses to third party source code, as stated in JLS-34.
- For software developed by us, what claims are we making, and how confident
are we about these claims?
- **Answer**: There is no software developed by us, as we just use the original nlohmann/json library within eclipse_score/inc_nlohmann_json. For the code developed in nlohmann/json, all claims and confidence are provided in the statements. See e.g., all the no-json-faults (NJF) items.
- **Answer**: There is no software developed by us, as we just use the original nlohmann/json library within eclipse_score/nlohmann_json. For the code developed in nlohmann/json, all claims and confidence are provided in the statements. See e.g., all the no-json-faults (NJF) items.

4 changes: 2 additions & 2 deletions TSF/trustable/assertions/TA-RELEASES_CONTEXT.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ to TT-RESULTS analysis.
**Evidence**

- list of reproducible SHAs
- **Answer**: JLS-14 ensures that the SHA value of the nlohmann/json library used within eclipse-score/inc_nlohmann_json coincides with the SHA value provided by Niels Lohmann (for the same version).
- **Answer**: JLS-14 ensures that the SHA value of the nlohmann/json library used within eclipse-score/nlohmann_json coincides with the SHA value provided by Niels Lohmann (for the same version).
- list of non-reproducible elements with:
- explanation and justification
- **Answer**: The only elements which are not reproducible are 8 CMake integration tests. The list of these, together with explanation and justification, is provided in TSF/docs/non_reproducible_tests.md.
Expand Down Expand Up @@ -70,7 +70,7 @@ R / (R + N + B + M / (M + X))
- **Answer**: We are very confident that all components are taken from within our controlled environment, as there are currently no external components used within the nlohmann/json library (as documented in JLS-34).
- How confident are we that all of the tools we are using are also under our
control?
- **Answer**: All tools used by nlohmann/json are mirrored within our controlled environment eclipse-score/inc_nlohmann_json. Therefore, these tools are under full control of the Eclipse S-Core organisation.
- **Answer**: All tools used by nlohmann/json are mirrored within our controlled environment eclipse-score/nlohmann_json. Therefore, these tools are under full control of the Eclipse S-Core organisation.
- Are our builds repeatable on a different server, or in a different context?
- **Answer**: Since there is no "build" of the header-only library, yes.
- How sure are we that our builds don't access the internet?
Expand Down
6 changes: 3 additions & 3 deletions TSF/trustable/assertions/TA-SUPPLY_CHAIN_CONTEXT.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,11 +41,11 @@ binaries present within the repositories of the components and tools used.
- without access to cached data
- **Answer**: The library and test suite build require no cached data. CMake configuration runs from scratch, generating all artifacts deterministically as shown in ubuntu.yml workflow clean builds.
- update logs for mirrored projects
- **Answer**: Updates of nlohmann/json are documented in release notes (see JLS-13). The update process of eclipse-score/inc_nlohmann_json is detailed in the "Update Concept" section of TSF/README.md.
- **Answer**: Updates of nlohmann/json are documented in release notes (see JLS-13). The update process of eclipse-score/nlohmann_json is detailed in the "Update Concept" section of TSF/README.md.
- mirrors reject history rewrites
- **Answer**: The eclipse-score/inc_nlohmann_json main branch is set up with a branch protection rule, which disallows force pushes and requires code reviews for all changes. The branch protection rule prevents history rewrites (see JLS-66).
- **Answer**: The eclipse-score/nlohmann_json main branch is set up with a branch protection rule, which disallows force pushes and requires code reviews for all changes. The branch protection rule prevents history rewrites (see JLS-66).
- mirroring is configured via infrastructure under direct control
- **Answer**: The eclipse-score/inc_nlohmann_json repository, which contains the mirrors, is configured by "Otterdog", which is under direct control by eclipse-score maintainers (see JLS-66).
- **Answer**: The eclipse-score/nlohmann_json repository, which contains the mirrors, is configured by "Otterdog", which is under direct control by eclipse-score maintainers (see JLS-66).

**Confidence scoring**

Expand Down
2 changes: 1 addition & 1 deletion TSF/trustable/assertions/TA-TESTS-CHECKLIST.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@

* How confident are we that all test components are taken from within our controlled environment?

**Answer**: All tests are either self-contained or download test data from [within Eclipse S-CORE](https://github.com/eclipse-score/inc_nlohmann_json/tree/json_test_data_version_3_1_0_mirror).
**Answer**: All tests are either self-contained or download test data from [within Eclipse S-CORE](https://github.com/eclipse-score/nlohmann_json/tree/json_test_data_version_3_1_0_mirror).

* How confident are we that all of the test environments we are using are also under our control?

Expand Down
2 changes: 1 addition & 1 deletion TSF/trustable/assertions/TA-TESTS_CONTEXT.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ tooling and their build environments are repeatable and reproducible.
- Are any exceptions identified, documented and justified?
- **Answer**: To the best of our knowledge, there are no exceptions identified.
- How confident are we that all test components are taken from within our controlled environment?
- **Answer**: All tests are either self-contained or download test data from [within Eclipse S-CORE](https://github.com/eclipse-score/inc_nlohmann_json/tree/json_test_data_version_3_1_0_mirror).
- **Answer**: All tests are either self-contained or download test data from [within Eclipse S-CORE](https://github.com/eclipse-score/nlohmann_json/tree/json_test_data_version_3_1_0_mirror).
- How confident are we that all of the test environments we are using are also under our control?
- **Answer**: Very confident, as the environments are standard docker images of ubuntu and standard versions of compilers that are executed in our CI pipeline.
- Do we record all test environment components, including hardware and infrastructure used for exercising tests and processing input/output data?
Expand Down
2 changes: 1 addition & 1 deletion TSF/trustable/assertions/TA-VALIDATION_CONTEXT.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ results for all expected tests (both pass / fail and performance).
- Is the selection of tests correct?
- **Answer**: The upstream nlohmann/json test suite plus the additional TSF-related tests cover the expected behaviours captured in the Expectations (JLEX-01, JLEX-02).
- Are the tests executed enough times?
- **Answer**: Yes, tests run on each pull request and push via the CI workflows (JLS-01) and additionally on a daily schedule in inc_nlohmann_json (JLS-22).
- **Answer**: Yes, tests run on each pull request and push via the CI workflows (JLS-01) and additionally on a daily schedule in nlohmann_json (JLS-22).
- How confident are we that all test results are being captured?
- **Answer**: For the CI workflows that write to the persistent test-results database we are reasonably confident, since capture is automated by the TSF scripts (JLS-18, JLS-45).
- Can we look at any individual test result, and establish what it relates to?
Expand Down
2 changes: 1 addition & 1 deletion TSF/trustable/assumptions-of-use/AOU-01.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,4 @@ level: 1.1
normative: true
---

The integrator shall report problems with eclipse-score/inc_nlohmann_json's implementation to the upstream nlohmann/json repository whenever a problem is detected.
The integrator shall report problems with eclipse-score/nlohmann_json's implementation to the upstream nlohmann/json repository whenever a problem is detected.
2 changes: 1 addition & 1 deletion TSF/trustable/assumptions-of-use/AOU-02.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,4 @@ level: 1.1
normative: true
---

The integrator shall ensure that the build environment used for eclipse-score/inc_nlohmann_json is supplied with consistent dependencies in every integrating system.
The integrator shall ensure that the build environment used for eclipse-score/nlohmann_json is supplied with consistent dependencies in every integrating system.
2 changes: 1 addition & 1 deletion TSF/trustable/assumptions-of-use/AOU-03.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,4 @@ level: 1.1
normative: true
---

The integrator shall ensure that integrator-controlled mirrors of the dependencies of the nlohmann/json repository are persistently and accessibly stored as long as the nlohmann/json library is used within eclipse-score/inc_nlohmann_json.
The integrator shall ensure that integrator-controlled mirrors of the dependencies of the nlohmann/json repository are persistently and accessibly stored as long as the nlohmann/json library is used within eclipse-score/nlohmann_json.
2 changes: 1 addition & 1 deletion TSF/trustable/assumptions-of-use/AOU-04.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,4 @@ level: 1.1
normative: true
---

The integrator shall ensure that exceptions are properly handled or turned off in eclipse-score/inc_nlohmann_json, whenever eclipse-score/inc_nlohmann_json's implementation of nlohmann/json is used.
The integrator shall ensure that exceptions are properly handled or turned off in eclipse-score/nlohmann_json, whenever eclipse-score/nlohmann_json's implementation of nlohmann/json is used.
2 changes: 1 addition & 1 deletion TSF/trustable/assumptions-of-use/AOU-05.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,4 @@ level: 1.1
normative: true
---

The integrator shall ensure that in eclipse-score/inc_nlohmann_json, input is encoded as UTF-8 (as required by RFC8259) and that in case other string formats are used, thrown exceptions are properly handled.
The integrator shall ensure that in eclipse-score/nlohmann_json, input is encoded as UTF-8 (as required by RFC8259) and that in case other string formats are used, thrown exceptions are properly handled.
2 changes: 1 addition & 1 deletion TSF/trustable/assumptions-of-use/AOU-07.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,4 @@ level: 1.1
normative: true
---

The integrator shall ensure in eclipse-score/inc_nlohmann_json that exceptions, which are expected during parsing with default parameters, are properly handled whenever the input is no valid JSON.
The integrator shall ensure in eclipse-score/nlohmann_json that exceptions, which are expected during parsing with default parameters, are properly handled whenever the input is no valid JSON.
2 changes: 1 addition & 1 deletion TSF/trustable/assumptions-of-use/AOU-08.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,4 @@ level: 1.1
normative: true
---

The integrator shall ensure that all necessary source files and built tools are mirrored in eclipse-score/inc_nlohmann_json, e.g. using a built server without internet access, as long as nlohmann/json is actively used within eclipse-score/inc_nlohmann_json.
The integrator shall ensure that all necessary source files and built tools are mirrored in eclipse-score/nlohmann_json, e.g. using a built server without internet access, as long as nlohmann/json is actively used within eclipse-score/nlohmann_json.
2 changes: 1 addition & 1 deletion TSF/trustable/assumptions-of-use/AOU-09.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,4 @@ level: 1.1
normative: true
---

The integrator shall ensure inside eclipse-score/inc_nlohmann_json that advanced warning indicators for misbehaviours are identified, and monitoring mechanisms are specified, verified and validated based on analysis.
The integrator shall ensure inside eclipse-score/nlohmann_json that advanced warning indicators for misbehaviours are identified, and monitoring mechanisms are specified, verified and validated based on analysis.
2 changes: 1 addition & 1 deletion TSF/trustable/assumptions-of-use/AOU-14.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,4 @@ level: 1.1
normative: true
---

The integrator shall ensure that the eclipse-score/inc_nlohmann_json is built with tools from the provided matrix specification, whenever nlohmann/json is used within eclipse-score/inc_nlohmann_json. (not yet provided)
The integrator shall ensure that the eclipse-score/nlohmann_json is built with tools from the provided matrix specification, whenever nlohmann/json is used within eclipse-score/nlohmann_json. (not yet provided)
Loading
Loading