Problematic SPARQL queries

[jhsoby]

For the current Norwegian contest (2025-03) and Finnish contest (2025-10), the bot intermittently fails because the SPARQL queries time out.

A potential fix for this would be to first compile the list of all articles participants have been contributing to during the contest, get their Wikidata IDs, and then check those articles (via something like [1]) against the query we want to check. This should be much faster, but would also require quite a few changes in how the bot does these queries.

[1]:

SELECT ?item WHERE {
  VALUES ?item { wd:Q57665 wd:Q291644 wd:Q133009 wd:Q207655 } .
  ?item wdt:P21 wd:Q6581072
}

[jcreusand]

Your suggestion would probably fix the current limitation that prevents contests from giving points to wikidata editions unless the criterion is a sparql query (at least, this is what I figured via Google translate from https://no.wikipedia.org/wiki/Mal:Ukens_konkurranse_poeng#wikidata

[jcreusand]

On the other hand, you're substituting one potentially huge query (all the women in wikipedia) for a few queries, one for each participant. If we have lots of participants (like hosting a contest in enwiki), this may escalate fast. And if some participant does like Theklan and sends 2.000 batch updates to Wikidata, we're toast again... We have to elaborate a little more on the problem ... Perhaps do some detection of the cost of the SPARQL query and, if it's too heavy, just cache it and update it once a day?

[zache-fi]

Just pickign errors to tickets

https://ukbot.toolforge.org/jobs/fi-vk_e8ee00a8-dbb8-434b-beee-2d7b248d5401

Running SPARQL query: 
            SELECT ?article19472065
            WHERE {
              { values ?ispartof { <https://fi.wikipedia.org/> <https://smn.wikipedia.org/> <https://olo.wikipedia.org/> <https://se.wikipedia.org/> } .
 hint:Query hint:optimizer "None". 
?article schema:isPartOf ?ispartof. 
?article schema:about ?item . 
?item wdt:P21 ?gender .  
MINUS {?item wdt:P21 wd:Q6581097} . 
MINUS { ?item wdt:P21 wd:Q44148 } }
              ?article19472065 schema:about ?item .
              ?article19472065 schema:isPartOf <https://se.wikipedia.org/> .
            }
        
Loading
Traceback (most recent call last):
  File "/data/project/ukbot/www/python/venv/lib/python3.11/site-packages/requests/models.py", line 971, in json
    return complexjson.loads(self.text, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
               ^^^^^^^^^^^^^^^^^^^^^^
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1194 column 5 (char 32768)

https://ukbot.toolforge.org/jobs/no-mk_fa75d3a8-0adc-4902-a6c3-82f2f07f3d62

Running SPARQL query: 
           SELECT ?article19472065
           WHERE {
             { { ?item wdt:P21 wd:Q6581072 } UNION { ?item wdt:P21 wd:Q1052281 } . }
             ?article19472065 schema:about ?item .
             ?article19472065 schema:isPartOf <https://commons.wikimedia.org/> .
           }
       
Loading
Traceback (most recent call last):
 File "/data/project/ukbot/www/python/venv/lib/python3.11/site-packages/requests/models.py", line 971, in json
   return complexjson.loads(self.text, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/usr/lib/python3.11/json/__init__.py", line 346, in loads
   return _default_decoder.decode(s)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/usr/lib/python3.11/json/decoder.py", line 337, in decode
   obj, end = self.raw_decode(s, idx=_w(s, 0).end())
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/usr/lib/python3.11/json/decoder.py", line 353, in raw_decode
   obj, end = self.scan_once(s, idx)
              ^^^^^^^^^^^^^^^^^^^^^^
json.decoder.JSONDecodeError: Expecting ':' delimiter: line 601802 column 25 (char 18096128)

[zache-fi]

I made a small PHP script for getting Wikidata id's

• https://fiwiki-tools.toolforge.org/get_wikidata_items_by_ukbot_competition.php?src=no&namespace=2&page=Zache/Sandkasse&start=20250300000000&target=commons

• https://fiwiki-tools.toolforge.org/get_wikidata_items_by_ukbot_competition.php?src=no&namespace=4&page=Konkurranser/Månedens_konkurranse_2025-03&start=20250300000000&target=no

Parameteres are

• src = source wiki (no, fi, ca is allowed)
• namespace = namespace number of competition page in source wiki
• page = page name of the competition page
• start = start timestamp in format YYYYMMDDHHMMSS
• end = end timestamp in format YYYYMMDDHHMMSS (OPTIONAL, DEFAULT = NOW + 1 DAY)
• target = target wiki where wikidata ids checked (no, fi, smn, se, sv, ca, commons is allowed)

Source code

With this we can test if SPARQL queries could be succesfully filtered by using VALUES item list.

[zache-fi]

If we want run SQL queries directly from Ukbot instead of creating custom APIs, but we do not want to create hard dependencies which would require either port forwarding or running code on toolforge then we can use Superset API for running queries. There is performance penalties in this aproach but there is example on how it is done in Pywikibot.

• https://phabricator.wikimedia.org/T367684

[zache-fi]

I updated also ukbot side code now. Note. large SPARQL queries will require using POST instead of GET or they will fail with 400 error. That was the reason why i changed from GET to POST.

• https://github.com/WikimediaNorge/UKBot/tree/29-problematic-sparql-queries

[zache-fi]

Just as a note: Updating the graph doesn't work if the initialization of the competition rules fails for some reason. For example, if the SPARQL query times out, then it doesn't update the graph either. This is a bigger problem in graph updates as they will run only once per day instead of every hour, so failures are much more visible.