Wikidata in NZ divisions missing?

[davelowndes]

I have been using the 2025-05-21.0 release. Around 85% (2989) of the 3498 divisions in 'NZ' had wikidata ID. Now there's only 18 in the 2026-01-21.0 release, for just the country and regions. The Wikidata ID is important for creating relationships to other sources. Do we know why the Wikidata IDs are missing?

docker run --rm duckdb/duckdb duckdb -c "
INSTALL httpfs; LOAD httpfs;
SET s3_region='us-west-2';
SELECT id, subtype, country, wikidata, names.primary AS name
FROM read_parquet('s3://overturemaps-us-west-2/release/2026-01-21.0/theme=divisions/type=division/*')
WHERE country='NZ' AND wikidata IS NOT NULL
ORDER BY subtype, name
;"

Is it expected that releases would drop Wikidata IDs like this, either in NZ or other countries? It doesn't make for a stable dataset if we're basing applications on this field.

[danabauer]

Hi @davelowndes, thanks for reaching out. Back in September, we switched to two new sources for NZ divisions data, including a local authoritative source. See the release notes. This could be part of the reason for the significant decrease in Wikidata IDs. One of my colleagues who works closely with divisions will respond here shortly with more detailed information.

Separately, I'm glad to hear you've moved to the latest release. This is exactly what we want people doing with Overture data. Did you face any other challenges as you migrated to the latest data and schema? And have you been using GERS IDs to map Overture data to other sources?

[davelowndes]

Hi @danabauer Thanks for the reply. I haven't noticed any other issues moving to the latest release. I'm not sure how I can map GERS to the wikidata source I need, although the GERS property is catalogued at https://www.wikidata.org/wiki/Property:P13219 I can't find any places in wikidata that have this property set.

[davelowndes]

BTW, I thought this is/was a bug so I created #454

[sasastanojkov]

Hey @davelowndes! Happy to see you are using the data we are producing.
With regards to wikidata issue. We migrated NZ divisions to Linz data source as it has better data (mainly polygons of localities and neighborhoods) than OpenStreetMap. In this transition we lost wikidata information as we prioritized geometry over wikidata info. Entity GERS ids are expected to be the same so if you have some of the previous drops you can match and use it for your use-case. GERS id should be the main property used for mapping across datasets, so we made sure to keep it stable. I don't think it would be applicable for us to add wikidata on top of current data as I am not sure the licensing allows us to mix and match like that. So, at this point I don't see us producing wikidata in NZ any time soon.
You are seeing it still for country/region because those are still OpenStreetMap data.
In the future, I can see us again prioritizing polygons and other properties over wikidata when finding and transitioning to new datasets, so I wouldn't rely too much on wikidata info being fully stable. Your safer bet is always GERS which is our top priority to keep stable.

[davelowndes]

@sasastanojkov Thank you for confirmation. Aside from a previous drop, is there a long-term mapping of GERS to OpenStreetMap IDs? I found the boundary files, and I can get the OSM ID of Auckland, NZ using the GERS ID, but I'm not sure if this will be available long-term? Considering the sources array in the division for Auckland does not have an entry for OSM. It doesn't look like the data at OSM is associating its nodes with GERS ID yet either. This will end up being a problem for my use case as GERS is limited in my ability to associate with multiple sources for data enrichment. Do we expect other sources like wikidata and OSM to eventually have widespread adoption of the GERS ID as metadata?

Query to get Bridge File records for Auckland, NZ:

docker run --rm duckdb/duckdb duckdb -c "
INSTALL httpfs; LOAD httpfs;
SET s3_region='us-west-2';
SET variable gers_id = 'c69a2614-72e3-49d0-af95-e25700aaf1b4';

SELECT
  id AS gers_id,
  dataset,
  theme,
  type,
  record_id,
  update_time
FROM read_parquet(
  's3://overturemaps-us-west-2/bridgefiles/2026-01-21.0/*/theme=divisions/type=division/*',
  hive_partitioning=1
)
WHERE id = getvariable('gers_id')
ORDER BY record_id
;"

┌──────────────────────────────────────┬───────────────┬───────────┬──────────┬───────────────┬──────────────────────┐
│               gers_id                │    dataset    │   theme   │   type   │   record_id   │     update_time      │
│               varchar                │    varchar    │  varchar  │ varchar  │    varchar    │       varchar        │
├──────────────────────────────────────┼───────────────┼───────────┼──────────┼───────────────┼──────────────────────┤
│ c69a2614-72e3-49d0-af95-e25700aaf1b4 │ Linz          │ divisions │ division │ 100004        │ 2021-01-18T00:00:00Z │
│ c69a2614-72e3-49d0-af95-e25700aaf1b4 │ OpenStreetMap │ divisions │ division │ n292806332@47 │ 2025-12-20T10:41:23Z │
└──────────────────────────────────────┴───────────────┴───────────┴──────────┴───────────────┴──────────────────────┘

[sasastanojkov]

I can partially answer.

@danabauer perhaps you can answer this:

Do we expect other sources like wikidata and OSM to eventually have widespread adoption of the GERS ID as metadata?

Bridge files are generally way to go for mapping different datasets to GERS id. In this case we have OSM -> GERS mapping and I would expect that the for forcible future it will be there since we do have OpenStreetMap NZ data in our corpus, we just choose not to release it as part of our drops. As to answer why it isn't in the source field, source field contains information about all sources that were used to create that one published entity. In case of Linz entities it should almost always only have single record referring to the Linz id.

There are >3k divisions with country = 'NZ'. And there are ~2k mappings of those entities in bridge files. I don't expect that OSM has all the same entities as Linz but I would expect that bridge files should be good enough for you to get most entity mappings OSM -> GERS even though data is coming from Linz. (I hope I didn't overcomplicate it :D)

Here is a duckdb sample code to get you started.

SELECT localities.*, bridge_files.record_id
FROM
(
	SELECT id, subtype, names.primary AS name
	FROM read_parquet('s3://overturemaps-us-west-2/release/2026-01-21.0/theme=divisions/type=division/part-00000-07df6208-78ed-480b-b593-2d19c61cfdef-c000.zstd.parquet')
	WHERE country = 'NZ'
) localities
INNER JOIN
(
	SELECT id, record_id
	FROM read_parquet('s3://overturemaps-us-west-2/bridgefiles/2026-01-21.0/dataset=OpenStreetMap/theme=divisions/type=division/*.parquet')
) bridge_files
ON bridge_files.id = localities.id

[davelowndes]

Thank you for the response @sasastanojkov. I do appreciate you taking the time.

The join to the bridge files produces ~2k OSM ids (which would lead to an unknown number of wikidata ids at OSM), so I do wonder how I got ~3k back in the 2025-05-21.0 release. I'm just trying to get an understanding of provenance and linkage, as in my honest opinion, being able to link overture data to external sources opens up a lot more opportunities and use cases, of which my app is one. So I'd hope overture would consider this in its long term strategy. If I can get a link to wikidata, this opens up links to all sorts of content, consider the Identifiers attached to https://www.wikidata.org/wiki/Q90 for instance.

For now I'll take your advice and use the 2025-05-21.0 references, lucky I have a backup!

[sasastanojkov]

I fully agree with you. And yea, we will revisit and try to come up with proper legal solution to open up wikidata information for sources other than OpenStreetMap.

Hope this mitigation helps you for now. Feel free to reach out for any additional info.
And yea, I think it is ok to close this discussion if you agree :D

[davelowndes]

Thanks very much @sasastanojkov 😊