Skip to content

Add parser performance benchmarks with CI comparison workflow#102

Draft
Copilot wants to merge 5 commits intomainfrom
copilot/add-parser-performance-tests
Draft

Add parser performance benchmarks with CI comparison workflow#102
Copilot wants to merge 5 commits intomainfrom
copilot/add-parser-performance-tests

Conversation

Copy link
Contributor

Copilot AI commented Jan 14, 2026

Restores performance testing for the parser using the ordo-one/package-benchmark framework, previously removed in #100.

Changes

  • Benchmark implementation: Two benchmarks using the existing 6.5k line Big.beancount test file

    • Parse Big File: Measures file I/O + parsing via Parser.parse(contentOf:)
    • Parse Big File (String): Measures pure parsing via Parser.parse(string:) with pre-loaded content
  • CI workflow: Runs on PRs, benchmarks base vs head in same runner, posts markdown comparison to PR, fails on >10% regression

  • Dependencies: Added package-benchmark 1.29.7+, includes jemalloc for memory profiling

let benchmarks = {
    let config = Benchmark.Configuration(
        metrics: [.wallClock, .throughput],
        scalingFactor: .kilo,
        maxDuration: .seconds(10)
    )

    Benchmark("Parse Big File", configuration: config) { benchmark in
        let fileURL = loadBigFile()
        benchmark.startMeasurement()
        for _ in benchmark.scaledIterations {
            _ = try Parser.parse(contentOf: fileURL)
        }
    }
}

Workflow follows the official CI comparison guide for baseline comparison in the same runner environment.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • esm.ubuntu.com
    • Triggering command: /usr/lib/apt/methods/https /usr/lib/apt/methods/https -parse-as-library -incremental -c @/home/REDACTED/work/SwiftBeanCount/SwiftBeanCount/.build/x86_64-REDACTED-linux-gnu/debug/SwiftBeanCountParser.build/sources -I /home/REDACTED/work/SwiftBeanCount/SwiftBeanCount/.build/x86_64-REDACTED-linux-gnu/debug/Modules rk/Sources/Benchmark/ARCStats/ARCStats.swift x86_64-REDACTED-linux-gnu rk/Sources/Benchmark/ARCStats/ARCStatsProducer.swift -enable-batch-mode rk/Sources/Benchmark/Benchmark+ConvenienceInitializers.swift -e rk/Sources/Benchmark/Benchmark.swift git rk/Sources/Benchmark/BenchmarkClock.swift l rk/Sources/Benchmark/BenchmarkExecutor+Extensions.swift 917bb95775729d4f-emit-module rk/Sources/Benchmark/BenchmarkExecutor.swift (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Original prompt

This section details on the original issue you should resolve

<issue_title>Parser performance tests</issue_title>
<issue_description>The parser had an attempt at performance tests, which were removed in #100
We should bring them back, using https://www.swift.org/blog/benchmarks/

for this we should follow https://swiftpackageindex.com/ordo-one/package-benchmark/1.29.7/documentation/benchmark/comparingbenchmarksci and probably use CI to run the base and head branch of a PR after each other in the same runner (but different action from the testing) to compare and fail at a certain threshold.</issue_description>

Comments on the Issue (you are @copilot in this section)


💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

Copilot AI and others added 4 commits January 14, 2026 09:02
- Added ordo-one/package-benchmark dependency to Package.swift
- Created SwiftBeanCountParserBenchmarks executable target
- Implemented two parser benchmarks: file parsing and string parsing
- Added Big.beancount test resource for benchmarks
- Updated .gitignore to exclude benchmark artifacts

Co-authored-by: Nef10 <4551135+Nef10@users.noreply.github.com>
- Created benchmarks.yml workflow to run on PRs
- Workflow compares base vs PR branch performance
- Configured 10% threshold for performance regression detection
- Added jemalloc installation step for Linux
- Fixed SwiftLint violations in benchmark code
- Benchmarks now lint cleanly

Co-authored-by: Nef10 <4551135+Nef10@users.noreply.github.com>
- Removed unnecessary availability check for String(contentsOf:encoding:)
- Use explicit UTF-8 encoding consistently
- Simplified code by removing conditional logic

Co-authored-by: Nef10 <4551135+Nef10@users.noreply.github.com>
- Added comment to clarify file loading happens before measurement
- This ensures only parsing performance is benchmarked, not I/O

Co-authored-by: Nef10 <4551135+Nef10@users.noreply.github.com>
Copilot AI changed the title [WIP] Add parser performance tests for benchmarking Add parser performance benchmarks with CI comparison workflow Jan 14, 2026
Copilot AI requested a review from Nef10 January 14, 2026 09:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Parser performance tests

2 participants