Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions .dotstop.dot
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,6 @@ digraph G {
"AOU-15" [sha=f4d2fdcacc3c3bd743af3ff5c2a08ceb650e59015ed846c35ddb4de104e80cec];
"AOU-16" [sha="4e08bac839cba01a4dc2a794bd2d1254d388945c6b79293684e00b9f8d03f31e"];
"AOU-17" [sha=ce26eef503179bbedca706f43fedc289d8c56580c4b885b3abab440d2de25870];
"AOU-18" [sha="672193d261861effb34f845d6e3421f47ce85e79d7ef34f62ce5c10bef68e796"];
"AOU-19" [sha="877c346448621a81b2d1418e610575bfcd556d2f56a14f23f289b90828907928"];
"AOU-20" [sha="d17ea5c6a3f07b516abc7c51d4b15b1a130a28602fbd4eb6872006a2e5f939d0"];
"AOU-21" [sha="d1712fcf7a0a00659021a69784f697b5d7b404aeadb7be66553218ba2e6c0bf7"];
Expand Down Expand Up @@ -91,6 +90,7 @@ digraph G {
"JLS-64" [sha="40f1382c156e308ee543c30df4dc7eb457ac14d472909c30eb6caae9a3bc1d68"];
"JLS-65" [sha="e413de6c831c1c019c67c3e3477b9dc9302cc79433ec894beaee0c95e053b545"];
"JLS-66" [sha="cf57eaf55654ef52589b1879c7294de13ddf1258ecdff4f6371178c6e8e6975b"];
"JLS-74" [sha="c161214f0f206f3c0826750978fcc4c99e2765a0c3333592e1293b323434ca34"];
"NJF-01" [sha="548dc86014e093974f68660942daa231271496a471885bbed092a375b3079bd8"];
"NJF-02" [sha="6ea015646d696e3f014390ff41612eab66ac940f20cf27ce933cbadf8482d526"];
"NJF-03" [sha="4bd1f8210b7bba9a248055a437f377d9da0b7576c5e3ed053606cf8b5b2febe3"];
Expand Down Expand Up @@ -400,6 +400,9 @@ digraph G {
"PJD-03" -> "PJD-02" [sha="34c3d9163590aec428486eef28fa3895a3c425b2ceb826d3c1d592d34eefe14e"];
"TA-ANALYSIS" -> "JLS-17" [sha=eac76c7d417f623d9dd1a220fc0b3ec6e2c2b9e16781eefae6d21dbd35d1402d];
"TA-ANALYSIS" -> "JLS-26" [sha="250c8c20314a242c1daac1278ef5a96d97bce8d943318430a91ee30580b5b29d"];
"TA-ANALYSIS" -> "JLS-74" [sha="438058a274824cc8774588099abf500dcac8fa0215c486f79a43e00c6baa66a4"];
"TA-ANALYSIS" -> "JLS-31" [sha="80b0be4318282eee6799e35ef618a077539ec0d4c783db2e50e68d00e9085018"];
"TA-ANALYSIS" -> "JLS-27" [sha="cb77a6a19e50c33a85cc9b099e4401321f7460a9a6afef31bd2b9da29fe063af"];
"TA-BEHAVIOURS" -> "JLEX-01" [sha="8cd931ef61b7012140344adf54469e943bfc690ee54f12db12777464880061db"];
"TA-BEHAVIOURS" -> "JLEX-02" [sha=cb26451e31a56b1eb51a4d45283ba4a7c6e898efbd045b59cba10d3c6aa093de];
"TA-BEHAVIOURS" -> "JLS-03" [sha=cf9211c07452914cb2d0b455f859b26cb2724423eae5187e8cbfdff06d1b5ba3];
Expand All @@ -423,7 +426,6 @@ digraph G {
"TA-CONSTRAINTS" -> "AOU-09" [sha="b47001a26392bd151e4a03feccb198c74fdfbdfaeb055fd682b59e6b92f0bed5"];
"TA-CONSTRAINTS" -> "AOU-15" [sha="936c697c984cfadd07db6e2eef5a3f5f540c57d099c7a773c00e551ea15bdf7a"];
"TA-CONSTRAINTS" -> "AOU-17" [sha="b9866189c44d25116cc7cd47432546f8ea2b3aee0b493b364a89872e11957a69"];
"TA-CONSTRAINTS" -> "AOU-18" [sha="ca8930b0d06ea3987a85112508f49cfaf1cf77d9a405535f3173975175f15d78"];
"TA-CONSTRAINTS" -> "AOU-19" [sha="ef2ca2a552493258538bf72c975d4c0ec711cea193adb804765feea86382ec0a"];
"TA-CONSTRAINTS" -> "AOU-22" [sha="54608fda93406fa3546c2d38ef09d34f7252e2c40bb6bc982decf762ab1c68f0"];
"TA-CONSTRAINTS" -> "AOU-23" [sha="6a9aa9e8b107c684e68b21b44c363185c5f0668c39307a00758781d65f3e61d4"];
Expand Down
4 changes: 2 additions & 2 deletions .dotstop_extensions/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -410,10 +410,10 @@ Note: Starting with trudag v2025.09.16, "Commit date/time" is a unix timestamp (

## push

This functionality writes the generated data into an sqlite database `TrustableScoring.db` located in the folder `TSF`. This database contains two tables, `commit_info`, where the metadata of "info" are stored, and `scores`, where the scores are stored, and which references `commit_info` via the date as foreign key.
This functionality writes the generated data into an sqlite database stored under TSF/data_storage/ (e.g. TrustableScoring_YYYY-MM-DD.db). The file path can be configured via the environment variable TSF_SCORING_DB. This database contains two tables, `commit_info`, where the metadata of "info" are stored, and `scores`, where the scores are stored, and which references `commit_info` via the date as foreign key.

It is intended to store data only once per commit. If, for any reason, the same commit generates data more than once, then only the most recent data are stored, and the obsolete data are deleted. This still ensures that the scoring history of the main branch is as complete as possible.

## pull

This functionality parses the information stored in `TrustableScoring.db` into the format which is expected by trudag. In case that no data is found, the empty history is returned.
This functionality parses the information stored in the scoring database configured via TSF_SCORING_DB (stored under TSF/data_storage/) into the format which is expected by trudag. In case that no data is found, the empty history is returned.
2 changes: 1 addition & 1 deletion .dotstop_extensions/data_store.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
import os

# global variable -- path to persistent data storage
persistent_storage = "TSF/TrustableScoring.db"
persistent_storage = os.getenv("TSF_SCORING_DB", "TSF/TrustableScoring.db")

def data_store_pull() -> list[dict]:
data = get_my_data()
Expand Down
64 changes: 53 additions & 11 deletions .github/workflows/publish_documentation.yml
Original file line number Diff line number Diff line change
Expand Up @@ -121,17 +121,58 @@ jobs:
git commit -m "Updated issues list" || echo "No changes to commit"
git push origin save_historical_data && git pull

- name: Load persistent data
- name: Select persistent scoring DB (rotate if near GitHub limit)
shell: bash
run: |
if ! git ls-tree --name-only origin/save_historical_data TSF/TrustableScoring.db | grep TSF/TrustableScoring.db; then
mkdir -p TSF
touch TSF/TrustableScoring.db
git add TSF/TrustableScoring.db
git commit -m "Initialise persistent data storage"
git push origin save_historical_data
set -euo pipefail

DB_DIR="TSF/data_storage"
DB_PREFIX="TrustableScoring"
LIMIT_BYTES=$((90 * 1024 * 1024)) # 90 MiB
today=$(date -u +%F)

echo "[info] Rotation threshold: ${LIMIT_BYTES} bytes"
echo "[info] Today (UTC): ${today}"

# newest dated DB in storage branch
latest="$(git ls-tree -r --name-only origin/save_historical_data "$DB_DIR" \
| grep -E "^${DB_DIR}/${DB_PREFIX}_[0-9]{4}-[0-9]{2}-[0-9]{2}(_[0-9]{3})?\.db$" \
| sort | tail -n 1 || true)"

mkdir -p "$DB_DIR"

if [ -z "${latest}" ]; then
latest="${DB_DIR}/${DB_PREFIX}_${today}.db"
sqlite3 "$latest" "PRAGMA user_version = 1;"
echo "[info] No existing scoring DB -> created: $latest"
echo "TSF_SCORING_DB=${latest}" >> "$GITHUB_ENV"
exit 0
fi
git checkout $branch_name
git checkout save_historical_data -- TSF/TrustableScoring.db

git checkout save_historical_data -- "$latest"
size=$(stat -c%s "$latest")
echo "[info] Current scoring DB: $latest (${size} bytes)"

if [ "$size" -ge "$LIMIT_BYTES" ]; then
base="${DB_DIR}/${DB_PREFIX}_${today}.db"
new="$base"
if [ -e "$new" ]; then
i=1
while [ -e "${DB_DIR}/${DB_PREFIX}_${today}_$(printf '%03d' "$i").db" ]; do
i=$((i+1))
done
new="${DB_DIR}/${DB_PREFIX}_${today}_$(printf '%03d' "$i").db"
fi
sqlite3 "$new" "PRAGMA user_version = 1;"
latest="$new"
echo "[info] Rotation triggered -> new DB: $latest"
else
echo "[info] Rotation not needed."
fi

echo "TSF_SCORING_DB=${latest}" >> "$GITHUB_ENV"
echo "[info] Using scoring DB: $latest"


- name: Generate trudag report
run: |
Expand All @@ -144,13 +185,14 @@ jobs:

- name: Checkout data storage branch
run: |
git stash push --include-untracked -- $(git status --porcelain | awk '{print $2}' | grep -v TSF/TrustableScoring.db)
git stash push --include-untracked -- $(git status --porcelain | awk '{print $2}' | grep -vE '^TSF/data_storage/TrustableScoring.*\.db$)
git checkout save_historical_data
git pull

- name: Store persistent data
run: |
git add TSF/TrustableScoring.db
echo "[info] Committing scoring DB: $TSF_SCORING_DB"
git add "$TSF_SCORING_DB"
git commit -m "Append data storage" || echo "Historical data already up to date."
git push origin save_historical_data

Expand Down
Loading