Skip to content

Conversation

@josill
Copy link

@josill josill commented Jan 29, 2026

Fix #270

Description

Adds spec_only query parameter to config list endpoints (/api/pipeline/config/sources, /api/pipeline/config/conditions, /api/pipeline/config/targets). When set to true, returns only the Spec field from each Config object, reducing payload size and enabling easier comparison of config specifications.

Key implementation details:

  • Database-level optimization using PostgreSQL jsonb_build_object('spec', config->'spec') to extract only the spec field at query time
  • Consistent boolean parameter parsing using strconv.ParseBool() (matching existing codebase pattern)
  • Backward compatible: default behavior unchanged when parameter is not set
  • Updated Swagger documentation for all three endpoints
  • Added GIN indexes on config JSONB columns (000007_add_config_jsonb_indexes) for optimal performance on containment queries and field extraction

Test

To test this pull request, you can run the following commands:

cd pkg/server
go test -v -run TestEndpoints

Manual testing:

# Full config (default)
curl "http://localhost:8080/api/pipeline/config/sources"

# Spec-only
curl "http://localhost:8080/api/pipeline/config/sources?spec_only=true"

Additional Information

Tradeoff

  • Database query complexity: Uses PostgreSQL JSONB functions instead of simple column selection, but provides significant bandwidth savings
  • Query building: Requires conditional query construction, adding some complexity to the database layer

Potential improvement

  • Consider adding similar optimization for other frequently filtered fields

@josill
Copy link
Author

josill commented Jan 29, 2026

There are some unrelated changes in the generated swagger files that were not generated from previous developments

@josill josill marked this pull request as ready for review January 29, 2026 13:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature Request: Return configs with similar spec

1 participant