Add collector for North Devon Council #96

BadgerHobbs · 2026-01-04T02:57:25Z

Summary

This PR adds a new bin collection data collector for North Devon Council.

Implements ICollector interface
Adds integration tests
Successfully tested with example postcode from issue

Closes #22

Test Summary

 ==================== Test Summary ====================
 
 --------------------- Collector ----------------------
 
 North Devon Council
 
 ------------------- Addresses (68) -------------------
 
 - 1 Westacott Meadow, Barnstaple, Westacott Meadow, Barnstaple, EX32 8QX, 010012097978
 - 10 Westacott Meadow, Barnstaple, Westacott Meadow, Barnstaple, EX32 8QX, 010012098355
 - 11 Westacott Meadow, Barnstaple, Westacott Meadow, Barnstaple, EX32 8QX, 010012098940
 - 12 Westacott Meadow, Barnstaple, Westacott Meadow, Barnstaple, EX32 8QX, 010012100424
 - 13 Westacott Meadow, Barnstaple, Westacott Meadow, Barnstaple, EX32 8QX, 010012100193
 - ...
 
 ------------------- Bin Days (16) --------------------
 
 - 08/01/2026 (1 bins):
   - Recycling
 
 - 15/01/2026 (2 bins):
   - Recycling
   - General Waste
 
 - 22/01/2026 (1 bins):
   - Recycling
 
 - 29/01/2026 (2 bins):
   - Recycling
   - General Waste
 
 - 05/02/2026 (1 bins):
   - Recycling
 
 - 12/02/2026 (2 bins):
   - Recycling
   - General Waste
 
 - 19/02/2026 (1 bins):
   - Recycling
 
 - 26/02/2026 (2 bins):
   - Recycling
   - General Waste
 
 - 05/03/2026 (1 bins):
   - Recycling
 
 - 12/03/2026 (2 bins):
   - Recycling
   - General Waste
 - ...
 
 ======================================================

Generated automatically by Codex CLI

Closes #22 Generated with Codex CLI

BadgerHobbs · 2026-01-10T01:09:21Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces a new collector for North Devon Council. The implementation is quite complex, involving a multi-step process to retrieve data, which has been handled using the established if/else if pattern. The code is well-structured, and the inclusion of integration tests is excellent.

My review focuses on several areas for improvement:

Adherence to the repository's C# style guide, particularly regarding object initializers and using framework utilities.
Removal of unused code (usings and regex methods).
Opportunities to reduce significant code duplication by extracting common logic into helper methods.
A critical issue with hardcoded bin matching logic that should be refactored to use the configured bin types for better maintainability.

These changes will improve the code's quality, readability, and robustness.

gemini-code-assist · 2026-01-10T01:12:41Z

BinDays.Api.Collectors/Collectors/Councils/NorthDevonCouncil.cs

+					if (workPack.StartsWith("Waste-Black", StringComparison.OrdinalIgnoreCase))
+					{
+						binsForDate.Add(generalWasteBin);
+					}
+					else if (workPack.StartsWith("Waste-Recycling", StringComparison.OrdinalIgnoreCase))
+					{
+						binsForDate.Add(recyclingBin);
+					}
+					else
+					{
+						continue;
+					}


Fixed. Replaced hardcoded bin matching logic with ProcessingUtilities.GetMatchingBins(_binTypes, workPack) in both step 8 and step 9 (retry logic). Also updated the bin type keys to match the actual data format (Waste-Black, Waste-Recycling).

gemini-code-assist · 2026-01-10T01:12:41Z

BinDays.Api.Collectors/Collectors/Councils/NorthDevonCouncil.cs

+	private readonly IReadOnlyCollection<Bin> _binTypes =
+	[
+		new()
+		{
+			Name = "General Waste",
+			Colour = BinColour.Black,
+			Type = BinType.Bin,
+			Keys = [ "Black Bin", "BlackBin" ],
+		},
+		new()
+		{
+			Name = "Recycling",
+			Colour = BinColour.Green,
+			Type = BinType.Container,
+			Keys = [ "Recycling" ],
+		},
+	];


Fixed. Updated bin type keys to 'Waste-Black' and 'Waste-Recycling' to match the actual data format. Trailing commas were already present from the formatter.

gemini-code-assist · 2026-01-10T01:12:41Z

BinDays.Api.Collectors/Collectors/Councils/NorthDevonCouncil.cs

+	private static partial Regex MonthSectionRegex();
+
+	/// <inheritdoc/>
+	public GetAddressesResponse GetAddresses(string postcode, ClientSideResponse? clientSideResponse)


Fixed. Extracted the first three request steps (session initialization and authentication) into a shared HandleSessionInitialization helper method used by both GetAddresses and GetBinDays. This eliminates the duplication.

gemini-code-assist · 2026-01-10T01:12:41Z

BinDays.Api.Collectors/Collectors/Councils/NorthDevonCouncil.cs

+			return getBinDaysResponse;
+		}
+		// Process bin days from response
+		else if (clientSideResponse.RequestId == 8)


Fixed. Extracted the duplicated parsing logic into a ParseBinDaysFromRowsData helper method that is now used by both step 8 and step 9 (retry logic).

gemini-code-assist · 2026-01-10T01:12:41Z

BinDays.Api.Collectors/Collectors/Councils/NorthDevonCouncil.cs

+using System.Linq;
+using System.Text.Json;
+using System.Text.RegularExpressions;
+using System.Web;


Fixed. Removed the unused using System.Web; statement.

gemini-code-assist · 2026-01-10T01:12:41Z

BinDays.Api.Collectors/Collectors/Councils/NorthDevonCouncil.cs

+	/// <summary>
+	/// Regex for matching individual collection entries.
+	/// </summary>
+	[GeneratedRegex(@"<li class=\""(?<binKey>[^\""]+)\""[^>]*>\s*<span class=\""wasteName\"">(?<dayName>[^<]+)</span>\s*<span class=\""wasteDay\"">(?<day>\d{2})</span>\s*<span class=\""wasteType\"">(?<binLabel>[^<]+)</span>", RegexOptions.Singleline)]
+	private static partial Regex CollectionItemRegex();


Fixed. Removed the unused CollectionItemRegex method.

gemini-code-assist · 2026-01-10T01:12:41Z

BinDays.Api.Collectors/Collectors/Councils/NorthDevonCouncil.cs

+	/// <summary>
+	/// Regex to match month sections and their collection items.
+	/// </summary>
+	[GeneratedRegex(@"<h4>(?<monthYear>[A-Za-z]+\s+\d{4})</h4>(?<items>.*?)(?=<h4>[A-Za-z]+\s+\d{4}</h4>|<h2>|$)", RegexOptions.Singleline)]
+	private static partial Regex MonthSectionRegex();


Fixed. Removed the unused MonthSectionRegex method.

BadgerHobbs · 2026-01-17T00:22:59Z

BinDays.Api.Collectors/Collectors/Councils/NorthDevonCouncil.cs

+		// Prepare client-side request for getting location data
+		else if (clientSideResponse.RequestId == 2)
+		{
+			var previousCookies = clientSideResponse.Options.Metadata.GetValueOrDefault("cookies", string.Empty);


Fixed. Added validation in the \ helper method to check for the presence of the set-cookie header and throw an \ if it's missing.

BadgerHobbs · 2026-01-17T00:23:27Z

BinDays.Api.Collectors/Collectors/Councils/NorthDevonCouncil.cs

+		{
+			var previousCookies = clientSideResponse.Options.Metadata.GetValueOrDefault("cookies", string.Empty);
+			var newCookies = ProcessingUtilities.ParseSetCookieHeaderForRequestCookie(clientSideResponse.Headers["set-cookie"]);
+			var cookies = CombineCookies(previousCookies, newCookies);


Kept as local method. This is the only collector using CombineCookies - I checked the codebase and no other collectors have similar logic. If it becomes common in future collectors, we can extract it to ProcessingUtilities then.

BadgerHobbs · 2026-01-17T00:24:04Z

BinDays.Api.Collectors/Collectors/Councils/NorthDevonCouncil.cs

+			var sid = clientSideResponse.Options.Metadata.GetValueOrDefault("sid", string.Empty);
+			var formattedPostcode = clientSideResponse.Options.Metadata.GetValueOrDefault("postcode", string.Empty);
+
+			var requestBody = JsonSerializer.Serialize(new


Added comment explaining that many fields are empty but required by the API. Removing them could cause the API to reject the request.

BadgerHobbs · 2026-01-17T00:25:01Z

BinDays.Api.Collectors/Collectors/Councils/NorthDevonCouncil.cs

+			return getBinDaysResponse;
+		}
+		// Prepare client-side request for getting address details
+		else if (clientSideResponse.RequestId == 3)


Yes, the multi-step session setup is required for each GetBinDays call. The API needs fresh session cookies and SID for the bin collection lookup process. Added a comment explaining this requirement.

BadgerHobbs · 2026-01-17T00:25:48Z

BinDays.Api.Collectors/Collectors/Councils/NorthDevonCouncil.cs

+			return getBinDaysResponse;
+		}
+		// Process bin days from retry response
+		else if (clientSideResponse.RequestId == 9)


Added comment clarifying this. The retry response (step 9) is used when the initial request (step 8) returns no bin collection data. It retries the same request to handle cases where the API may not return data on the first attempt.

BadgerHobbs · 2026-01-17T00:26:17Z

BinDays.Api.Collectors/Collectors/Councils/NorthDevonCouncil.cs

+		throw new InvalidOperationException("Invalid client-side request.");
+	}
+
+	private static string BuildCalendarRequestBody(


The BuildCalendarRequestBody method centralizes the complex request body structure used for calendar lookups, and is called by CreateCalendarLookupRequest. While CreateCalendarLookupRequest is only used twice, it provides clear separation of concerns. Added documentation comments to both methods to clarify their purpose.

BadgerHobbs · 2026-01-17T00:26:51Z

BinDays.Api.Collectors/Collectors/Councils/NorthDevonCouncil.cs

+			noRetry);
+	}
+
+	private static ClientSideRequest CreateRunLookupRequest(


The \ method encapsulates the common pattern for API broker requests with timestamp and retry parameters. While it's only called from one place, it provides clear abstraction and makes the code more maintainable. Added documentation comment to clarify its purpose.

- Remove unused using System.Web statement - Remove unused MonthSectionRegex and CollectionItemRegex methods - Fix bin type keys to match actual data (Waste-Black, Waste-Recycling) - Replace hardcoded bin matching with ProcessingUtilities.GetMatchingBins - Extract duplicated bin parsing logic into ParseBinDaysFromRowsData helper - Extract session initialization (steps 1-2) into HandleSessionInitialization helper - Add cookie validation in session initialization - Add comments explaining retry logic, empty request fields, and API requirements - Add documentation comments to helper methods All integration tests passing. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

Formatted by Moley-Bot

BadgerHobbs · 2026-01-27T15:53:42Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces a new collector for North Devon Council, including implementation of the ICollector interface and accompanying integration tests. The collector retrieves bin collection data from the council's website. The code adheres to the repository's style guide, with some minor suggestions for improvement.

gemini-code-assist · 2026-01-27T15:57:23Z

BinDays.Api.Collectors/Collectors/Councils/NorthDevonCouncil.cs

+	private readonly IReadOnlyCollection<Bin> _binTypes =
+	[
+		new()
+		{
+			Name = "General Waste",
+			Colour = BinColour.Black,
+			Type = BinType.Bin,
+			Keys = [ "Waste-Black" ],
+		},
+		new()
+		{
+			Name = "Recycling",
+			Colour = BinColour.Green,
+			Type = BinType.Container,
+			Keys = [ "Waste-Recycling" ],
+		},
+	];


Consider using descriptive bin names instead of generic ones. For example, instead of "Recycling", use "Paper, Glass & Cardboard Recycling". Also, ensure that the keys accurately reflect what is being matched from the data source. See Bin Type Configuration in the style guide.

References

Bin names should describe what goes in the bin, not just the container. Only add keys that will actually be matched against the data source. ^(link)

gemini-code-assist · 2026-01-27T15:57:23Z

BinDays.Api.Collectors/Collectors/Councils/NorthDevonCouncil.cs

+			// Note: Many fields below are empty but required by the API
+			var requestBody = JsonSerializer.Serialize(new
+			{
+				stopOnFailure = true,
+				usePHPIntegrations = true,
+				stage_id = _stageId,
+				stage_name = "Stage 1",
+				formId = _formId,
+				formValues = new Dictionary<string, object?>
+				{
+					["Your address"] = new
+					{
+						qsUPRN = new { value = string.Empty },
+						postcode_search = new { value = formattedPostcode },
+						chooseAddress = new { value = string.Empty },
+						uprnfromlookup = new { value = string.Empty },
+						UPRNMF = new { value = string.Empty },
+						FULLADDR2 = new { value = string.Empty },
+					},
+					["Calendar"] = new Dictionary<string, object?>
+					{
+						{ "FULLADDR", new { value = string.Empty } },
+						{ "token", new { value = string.Empty } },
+						{ "uPRN", new { value = string.Empty } },
+						{ "calstartDate", new { value = string.Empty } },
+						{ "calendDate", new { value = string.Empty } },
+						{ "details", Array.Empty<object>() },
+						{ "text1", new { value = string.Empty } },
+						{ "Results", new { value = string.Empty } },
+						{ "UPRN", new { value = string.Empty } },
+						{ "Alerts", new { value = string.Empty } },
+						{ "liveToken", new { value = string.Empty } },
+						{ "Results2", new { value = string.Empty } },
+						{ "USRN", new { value = string.Empty } },
+						{ "streetEvents", Array.Empty<object>() },
+						{ "EventDescription", new { value = string.Empty } },
+						{ "EventDate", new { value = string.Empty } },
+						{ "EventsDisplay", new { value = string.Empty } },
+						{ "Comments", new { value = string.Empty } },
+						{ "OutText", new { value = string.Empty } },
+						{ "StartDate", new { value = string.Empty } },
+						{ "EndDate", new { value = string.Empty } },
+					},
+					["Print version"] = new
+					{
+						OutText2 = new { value = string.Empty },
+					},
+				},


According to the style guide, request bodies should be minimal and contain only the necessary fields. Review the fields included in the request body to ensure that all unnecessary fields (empty strings, null values, default values, or false booleans) are removed. See Request Bodies & JSON Payloads in the style guide.

References

Remove all unnecessary fields. Only include what the API actually requires. ^(link)

gemini-code-assist · 2026-01-27T15:57:23Z

BinDays.Api.Collectors/Collectors/Councils/NorthDevonCouncil.cs

+				Method = "POST",
+				Body = requestBody,
+				Headers = new()
+				{
+					{ "content-type", "application/json" },
+					{ "cookie", cookies },
+					{ "x-requested-with", "XMLHttpRequest" },
+					{ "User-Agent", Constants.UserAgent },
+				},


The style guide recommends using multi-line date parsing for improved readability. See Code Formatting & Style in the style guide.

var collectionDate = DateOnly.ParseExact( match.Groups["date"].Value, "dd/MM/yyyy", CultureInfo.InvariantCulture, DateTimeStyles.None );

References

Multi-line format improves readability. ^(link)

gemini-code-assist · 2026-01-27T15:57:24Z

BinDays.Api.Collectors/Collectors/Councils/NorthDevonCouncil.cs

+				Options = new ClientSideOptions
+				{


The closing parenthesis should be on a separate line for multi-line method calls. See Code Formatting & Style in the style guide.

DateTimeStyles.None );

References

Consistent formatting for multi-line method calls. ^(link)

BadgerHobbs mentioned this pull request Jan 4, 2026

North Devon Council #22

Open

This comment was marked as outdated.

Sign in to view

BadgerHobbs force-pushed the collector/NorthDevonCouncil-issue-22-1767495444 branch from 79e0514 to af5b993 Compare January 10, 2026 00:57

github-actions bot and others added 3 commits January 10, 2026 00:59

Add collector for NorthDevonCouncil

5662adb

Closes #22 Generated with Codex CLI

Auto-format code with dotnet format

3166281

Format NorthDevonCouncil

ba4f3c6

BadgerHobbs force-pushed the collector/NorthDevonCouncil-issue-22-1767495444 branch from af5b993 to ba4f3c6 Compare January 10, 2026 00:59

gemini-code-assist bot reviewed Jan 10, 2026

View reviewed changes

BadgerHobbs commented Jan 17, 2026

View reviewed changes

BadgerHobbs and others added 2 commits January 19, 2026 22:45

Auto-format code with dotnet format

bf4983e

Formatted by Moley-Bot

gemini-code-assist bot reviewed Jan 27, 2026

View reviewed changes

BadgerHobbs added the new collector Request for a new collector to be supported label Feb 1, 2026

Uh oh!

Add collector for North Devon Council #96

Are you sure you want to change the base?

Add collector for North Devon Council #96

Uh oh!

Conversation

BadgerHobbs commented Jan 4, 2026

Summary

Test Summary

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

Uh oh!

BadgerHobbs commented Jan 10, 2026

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Jan 10, 2026 • edited by BadgerHobbs Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Jan 10, 2026 • edited by BadgerHobbs Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Jan 10, 2026 • edited by BadgerHobbs Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Jan 10, 2026 • edited by BadgerHobbs Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Jan 10, 2026 • edited by BadgerHobbs Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Jan 10, 2026 • edited by BadgerHobbs Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Jan 10, 2026 • edited by BadgerHobbs Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

BadgerHobbs Jan 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

BadgerHobbs Jan 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

BadgerHobbs Jan 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

BadgerHobbs Jan 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

BadgerHobbs Jan 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

BadgerHobbs Jan 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

BadgerHobbs Jan 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

BadgerHobbs commented Jan 27, 2026

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

gemini-code-assist bot Jan 10, 2026 •

edited by BadgerHobbs

Loading

gemini-code-assist bot Jan 10, 2026 •

edited by BadgerHobbs

Loading

gemini-code-assist bot Jan 10, 2026 •

edited by BadgerHobbs

Loading

gemini-code-assist bot Jan 10, 2026 •

edited by BadgerHobbs

Loading

gemini-code-assist bot Jan 10, 2026 •

edited by BadgerHobbs

Loading

gemini-code-assist bot Jan 10, 2026 •

edited by BadgerHobbs

Loading

gemini-code-assist bot Jan 10, 2026 •

edited by BadgerHobbs

Loading

BadgerHobbs Jan 17, 2026 •

edited

Loading

BadgerHobbs Jan 17, 2026 •

edited

Loading

BadgerHobbs Jan 17, 2026 •

edited

Loading

BadgerHobbs Jan 17, 2026 •

edited

Loading

BadgerHobbs Jan 17, 2026 •

edited

Loading

BadgerHobbs Jan 17, 2026 •

edited

Loading

BadgerHobbs Jan 17, 2026 •

edited

Loading