[BUG] 429 not being ignored

[mhalano]

NOTE: IF YOU'RE UNSURE, PLEASE LOOK AT OR OPEN A DISCUSSION INSTEAD! https://github.com/mborsetti/webchanges/discussions

Describe the bug.
A clear and concise description of what the bug is. If it's related to an error, please also attach the traceback
of the exception obtained by running webchanges with -v or -vv.

When I get a 429 response code, webchanges still consider that the content was removed, which is a problem since it couldn't fetch the data in the first place. I'm ignoring response 4xx response codes.

erro-429.txt

Steps to reproduce.
Include relevant snippets of anything required (e.g. jobs and configuration files) for anyone to fully reproduce the
bug.

---
name: Support for upload of profiles mappings on Xbox Elite Series 2 wireless controller with xpadneo driver
url: https://raw.githubusercontent.com/atar-axis/xpadneo/master/docs/README.md
filters:
  - html2text: strip_tags
  - re.findall: 'profile mappings.*not supported'
  - striplines:

I have this line in my config file under job_defaults.url: ignore_http_error_codes: 4xx, 5xx

Expected behavior.
A clear and concise description of what you expected to happen.
Shouldn't trigger since it couldn't fetch the data, so it's impossible to define if the content really changed.

Screen scrape/screenshots.
If applicable, add screen scrape or screenshots to help explain the bug; please use -v or -vv to capture
logging.

Version info.
Please run webchanges --detailed-versions and paste the output here.

Software:
• webchanges: 3.31.4
• CPython: 3.13.7 ('main', 'Aug 20 2025 22:17:40') GCC 15.2.0
• SQLite: 3.46.1

System:
• Platform: Linux-6.17.0-6-generic-x86_64-with-glibc2.42, x86_64
• Processor: 
• CPUs (logical): 16
• Free memory: 31.9G physical plus 0.0B swap.
• Free disk '/': 216.7G (48.8%)
• --max-threads default: 20

Relevant PyPi packages:
• beautifulsoup4: 4.13.4
• beautifulsoup4: 4.13.4
• colorama: 0.4.6
• cryptography: 43.0.0
  - certifi: 2025.1.31
  - mypy: 1.18.1
  - click: 8.1.8
• cssselect: 1.3.0
• h2: 4.3.0
• html2text: 2025.4.15
• html5lib: 1.1
  - six: 1.17.0
  - webencodings: 0.5.1
  - chardet: 5.2.0
  - lxml: 6.0.1
  - chardet: 5.2.0
  - lxml: 6.0.1
• httpx: 0.28.1
  - anyio: 4.11.0
  - certifi: 2025.1.31
  - idna: 3.10
• jq: 1.9.1
• keyring: 25.6.0
  - jaraco.classes: 3.4.0
  - jaraco.functools: 4.1.0
  - jaraco.context: 6.0.1
• lxml: 6.0.1
• markdown2: 2.5.4
• msgpack: 1.1.1
• packaging: 25.0
• Pillow: 11.3.0
• Pillow: 11.3.0
• platformdirs: 4.4.0
• playwright: 1.52.0
• psutil: 7.1.0
  - setuptools: 78.1.1
  - packaging: 25.0
  - requests: 2.32.5
  - virtualenv: 20.31.2
  - wheel: 0.46.1
  - wheel: 0.46.1
  - setuptools: 78.1.1
  - wheel: 0.46.1
• pyyaml: 6.0.3
• requests: 2.32.5
• tzdata: 2025.2
• vobject: 0.9.9
  - pytz: 2025.2
  - six: 1.17.0
• webchanges: 3.31.4
  - cssselect: 1.3.0
  - h2: 4.3.0
  - html2text: 2025.4.15
  - httpx: 0.28.1
  - lxml: 6.0.1
  - markdown2: 2.5.4
  - msgpack: 1.1.1
  - platformdirs: 4.4.0
  - pyyaml: 6.0.3
• zstandard: 0.24.0

Playwright browser:
• Name: chromium
• Version: 141.0.7390.122
• Free memory with browser loaded: 31.8G physical plus 0.0B swap

Installed dpkg dependencies:
• jq
   - jq: 1.8.1-3ubuntu1
• playwright
   - google-chrome-stable: 141.0.7390.122-1

Additional context.
Add any other context about the problem.

[github-actions]

🤖 Hi @mhalano, I've received your request, and I'm working on it now! You can track my progress in the logs for more details.

[mborsetti]

Hi @mhalano,

Thank you for your contribution.

The current default behavior, which alerts the user when monitoring fails, is the correct approach. This proactive notification is crucial because it informs the user that a potential change in the source might have been missed. For instance, if a user is monitoring a critical resource that triggers alarms, a notification about a monitoring failure is preferable to silence. This allows the user to address the monitoring issue promptly and ensure the integrity of their alerts.

Furthermore, providing users with the flexibility to disregard specific errors through the ignore_http_error_codes directive is a valuable option. This empowers users to tailor the monitoring behavior to their specific needs, allowing them to suppress alerts for known or accepted issues while still being notified of unexpected failures.

[mhalano]

@mborsetti sorry, I didn't understand. Why it's not respecting the 429 error? If the server returns that, shouldn't the server ignore instead of reporting the all content is gone, since it couldn't get the actual content to do the comparison? I didn't have this problem with 3.30 version.

[mborsetti]

@mhalano Oh, I see now. Thank you for the follow-up. I initially misunderstood and thought the issue was being notified about encountering a 429, which should happen by default. I now understand that the problem is with the content being treated as empty, causing a diff to be created (and an opposite diff when the error is no longer encountered) which, as you pointed out, should not happen.

This is likely related to #104, and am already refactoring the error-handling code. I will ensure this issue is also resolved before the next release.

Thanks for bringing this to my attention!

[github-actions]

🤖 Hi @mborsetti, I've received your request, and I'm working on it now! You can track my progress in the logs for more details.

[mborsetti]

The fix to #104, incorporated in the just-released version 3.21.0, fixed this as well.

A new directive to ignore transient network errors inspired by this bug report to come in the (hopefully near) future.