Skip to content

Conversation

@eatyourpeas
Copy link
Member

@eatyourpeas eatyourpeas commented Dec 25, 2025

Overview

This PR moves the census-platform from just calculating IMD and serving lists of LSOAs/LADs etc, to serving geometry files.

ONS supports high res and low res boundary geometry (shape files) of LSOAs, SOAs, Data Zones, either clipped to low tide, or including the peripheral islands. The high definition files are over 1GB so cannot be stored on github, and would take ages to load and serve.

The secret after much research is to serve the data from CrunchyData's pg_tileserv which is hosted as a separate container. This generates tiles which serve quickly and work together with a CDN which caches the tiles so they can be served first with minimal latency. And of course this uses geography so we will need to upgrade to postgis.

This PR therefore:

  1. updates postgres to postgis and installs all the deps such as gdal
  2. update the seed function to pull in all the full detail (BFC) geometry data from arcgis in chunks, then optimize it for serving and link it to the IMD models (Ireland is not available on arcgis so we have to wget the geojson and simplify the oas into SOAs)
  3. adds tests for the seed - this will take at least an hour to seed the database from scratch
  4. seeding involves downloading the data in batches into a temp table and then merging this in to the existing tables on the region codes, linked to the IMD tables so all the imd ranks and deciles are available for mapping.
  5. adds a new API endpoint which serves the detail
  6. will need a CDN in front of the APIM to return cached tiles or reroute to the pg_tileserv
  7. will need a new pg_tileserv container - there is currently a dockerfile and docker-compose for local dev to run the 3 containers (postgis, drf and pg_tileserv)
image image image

GIthub Pages
image

I am sorry about the size of the PR. I initially committed a number of large files which i failed to remove from the previous commits and in error wound the whole branch back to the initial commit.

pacharanero and others added 30 commits March 30, 2023 14:28
/validate does the same underlying database lookup at their end as the normal API so we don't need the additional call
…dation

Remove redundant postcode validation
Signed-off-by: anchit-chandran <anchit97123@gmail.com>
Signed-off-by: anchit-chandran <anchit97123@gmail.com>
`actions/{upload|download}-artifact` has bumped to `v4`

but `azure/webapps-deploy` needs to stay `v2`
Signed-off-by: anchit-chandran <anchit97123@gmail.com>
Signed-off-by: anchit-chandran <anchit97123@gmail.com>
@eatyourpeas eatyourpeas merged commit f5a2ca0 into live Jan 18, 2026
1 check passed
@eatyourpeas eatyourpeas deleted the shapes branch January 18, 2026 10:07
@rcpch-prout
Copy link

rcpch-prout bot commented Jan 18, 2026

Overdue on LIVE (merged by @eatyourpeas 10 minutes and 3 seconds ago) What's gone wrong?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants