Searching for names containing apostrophes is unforgiving

[davecraig]

Hi!

I'm using Photon to provide the geo search within our application for blind and visually impaired users and one of the things that I've noticed is that places with apostrophes are harder to search for than places without.

It's not always easy to know whether there should be an apostrophe or not - one example of this is Princes Street in Edinburgh which was originally called Prince's Street but had the apostrophe dropped. Unless you've seen it written down it's guesswork to get it right (I actually thought that it was named Princess Street...).

The good thing is that searching for "Prince's Street" does return "Princes Street":
https://photon.komoot.io/api/?q=Prince%27s%20Street,Edinburgh

However, in the case where there is an apostrophe in the name, e.g. "Queen's View" (https://www.openstreetmap.org/node/71235429) searching without the apostrophe does not find it:
https://photon.komoot.io/api/?q=Queens%20View,Stirling fails
https://photon.komoot.io/api/?q=Queen%27s%20View,Stirling succeeds

Until now I hadn't looked at the code, so please excuse my ignorance, but I'm guessing that punctuationgreedy is what helps with dropping apostrophes when searching? What's interesting about that is that the apostrophe is replaced with a space. I experimented further and here's what works and doesn't work:

search	result
Queen View,Stirling	success
Que View,Stirling	success
Queen's View,Stirling	success
Queens View,Stirling	fail
Queens' View,Stirling	fail
Queend View,Stirling	fail

Based on this a workaround in English might be for me to knock of the last s in all search strings, though there are other cases where that wouldn't help e.g. "Bene't Street" in Cambridge. Searching on "Benet Street,Cambridge" doesn't succeed, though "Bene Street,Cambridge" does.

I'm interested in any suggestions as to how to look at this further.

Many thanks,

Dave Craig.

[lonvia]

I thought there is already an issue about that but looks like I mixed that up with Nominatim issues.

Long story short, apostrophes are easy in a monolingual context but a bit of a pain when you don't know what language the input is. OpenSearch has a token filter for English-style apostrophes but is very clear that this shouldn't be used on languages like French and Turkish. For French-style apostrophes there is the elision filter which in turn will cause havoc with English-style apostrophes. So sadly, there is no out-of-the-box solution here.

For single letter elisions, we might get away with just deleting the apostrophe without replacing it with a space. Then let fuzzy search do the rest at the expense of getting slightly worse results occasionally. However, this will not a workable approach for in-word apostrophes: fuzziness cannot add/delete word breaks.

[davecraig]

Thanks for taking the time to explain this, much appreciated.

I'm adding this extra explanation for anyone who stumbles across this discussion (and so I remember it later):

Photon initially attempts a search with a flag lenient set to false. This attempts an exact match of the string. If that fails to get any results, then the search is repeated with lenient set to true which results in a fuzzy search where a number of character edits are allowed to the search string (see https://docs.opensearch.org/latest/query-dsl/term/fuzzy/).

The search:
https://photon.komoot.io/api/?q=queens%20view,Stirling&lon=-4.4&lat=56.0&zoom=10&location_bias_scale=0.5
returns Stirling View in Queenstown, Singapore. That's a straight non-fuzzy search and because there's a result (an exact match), even though it's a long way away, that result is returned. No amount of lon/lat/zoom/bias tweaking will get rid of that result - though adding a bounding box does.

With a small-ish bounding box:
https://photon.komoot.io/api/?q=queens%20view,Stirling&bbox=-4.572732,55.814096,-3.995950,56.090125.
There are no exact results within the box and so a fuzzy search takes place and our target is included in the results. If the box was large enough that it included Stirling, then there are way more matches and the one that we are after isn't included.

If I change the s in queens to an a:
https://photon.komoot.io/api/?q=queena%20view,Stirling&lon=-4.4&lat=56.0&zoom=10&location_bias_scale=0.5
then there's no initial exact match and so a fuzzy search does take place and there are many more results including our target one.

The summary is, there is an apostrophe issue, but there are also too many things named after Queen Victoria and Views. Limiting the search area by using a bounding box greatly improves the chances of success.