Large wiki pages

[pnorman]

A few pages on the wiki are too large and complex for Mediawiki to properly handle them. This has in the past lead to downtime when they have stopped working and brought the entire wiki down with them.

The problems seem to be coming from pages with hundreds of images from Wikimedia Commons and high complexity pages with thousands of transcludes. The most prominent example is the Map features page which requires about 30 seconds to generate, but there are other examples which are worse for complexity.

In the case of map features, clicking on the edit link takes you to https://wiki.openstreetmap.org/w/index.php?title=Map_features&veaction=edit which, if it does not crash the browser, will time out.

On the complexity side, the number of images causes Wikimedia commons to consider the requests a DOS attack and block the user. This happens when the user views the page, but also when various scripts are run.

We've tried papering over the issues as they've come up but we need to address the root cause: Mediawiki does not work with pages this complex or with this many images.

By viewing source you can see a report at the bottom of the page in a HTML comment - search for NewPP limit report. For map features, it shows that has 5000+ transclusions and took 28.7s to generate when I loaded the page. For comparison, the front page has about 50 transclusions and takes 0.69 seconds to generate.

Wikipedia's page on their template limits is a reasonable overview of some of the problems.

Another class of page with problems is lists of thousands of relations.

I think there's a few steps we need to do.

1. We should start by telling wiki admins that some pages have become excessively complex and must be reduced. We can let them figure out what to do with the pages. Some of them like lists of thousands of relations may not belong on the wiki, while others should have been split some time ago for editorial reasons.
2. Tweak the various mediawiki parameters in an incremental manner, bringing them down to a level that prevents pages so complex they pose a risk to the entire wiki

[tomhughes]

This also caused problems when we added mod_evasive based rate limiting because pages like map features generate hundreds of requests and HTTP/2 allows a browser to make all those requests (up to about 600 for map features) within a a few seconds.

[pnorman]

See also #466.

Do we have a good way to identify the pages that need reducing in complexity?

Some image-heavy pages in the previous issue were

• https://wiki.openstreetmap.org/wiki/MUTCD/R
• https://wiki.openstreetmap.org/wiki/Key:maxweight
• https://wiki.openstreetmap.org/wiki/Ohio/Map_features
• https://wiki.openstreetmap.org/wiki/United_States_roads_tagging/Routes
• https://wiki.openstreetmap.org/wiki/Tag:boundary%3Dadministrative
• https://wiki.openstreetmap.org/wiki/Past_events_2020

Checking the reports for these was a pain because you have to wait for the page to load, and all of them are slow loading since they're part of the problem.

Pages like https://wiki.openstreetmap.org/wiki/Featured_images/Jan-Mar_2019 do not appear to be a problem.

Map Features is perhaps a worst-case scenario because it is both excessively complex and triggers too many requests.

[matkoniecz]

Do we have a good way to identify the pages that need reducing in complexity?

category such as https://wiki.openstreetmap.org/wiki/Category:Pages_where_node_count_is_exceeded is mentioned in mediawiki documentation but it is empty for some reason on OSM Wiki (maybe it needs to be configured? maybe it is hit by the bug that caused categories to be not filled?)

It works at https://en.wikipedia.org/wiki/Category:Pages_where_node_count_is_exceeded

[pnorman]

category such as https://wiki.openstreetmap.org/wiki/Category:Pages_where_node_count_is_exceeded is mentioned in mediawiki documentation but it is empty for some reason on OSM Wiki (maybe it needs to be configured? maybe it is hit by the bug that caused categories to be not filled?)

I would expect that is for the hard limits on the server which are too high right now.

Two others special categories that do list some items are
https://wiki.openstreetmap.org/wiki/Category:Pages_where_template_include_size_is_exceeded
https://wiki.openstreetmap.org/wiki/Category:Pages_with_too_many_expensive_parser_function_calls

[1ec5]

On the complexity side, the number of images causes Wikimedia commons to consider the requests a DOS attack and block the user. This happens when the user views the page, but also when various scripts are run.

Wikimedia servers will throttle requests from OSM – it even does so while viewing some large categories on Wikimedia Commons itself. But I was unaware of Wikimedia blocking OSM’s IP address. Is that documented somewhere?

[pnorman]

Wikimedia servers will throttle requests from OSM – it even does so while viewing some large categories on Wikimedia Commons itself. But I was unaware of Wikimedia blocking OSM’s IP address. Is that documented somewhere?

There are scripts that are supposed to run on the pages but we've had to turn off

[1ec5]

A server-side script like UpdateSpecialPages.php, or a client-side script like a gadget or site script?

[tomhughes]

I assume he's referring to us recently dropping a job that ran refreshLinks.php weekly after spending some months trying and failing to get it to run reliably.

One problem was that it leaks memory like a sieve, which is why the mediawiki manual advises running it in chunks and gives a script to do so but I found that even that was hanging, seemingly while trying to talk to commons, presumably when it was refreshing commons links.

[1ec5]

Some of the pages listed in #1046 (comment) can probably be split up or replaced with off-wiki resources. I’ve been trying to interest the community in replacing the “Map features” page with a dynamic off-wiki page based on id-tagging-schema: openstreetmap/id-tagging-schema#646. We could migrate large key pages, such as the one for cuisine=*, to taginfo taglists or just remove the table of values in favor of a link to the existing category.

The sign catalogs are tricky because their sole purpose is to annotate Commons images with community-developed tagging suggestions. MediaWiki’s links table is important because it lets readers know which tags are documented or undocumented and notifies proposal authors (via Echo) that their tags are being adopted by the community. Long-term, these catalogs should also live outside the wiki via either osmlab/name-suggestion-index#8225 or openstreetmap/id-tagging-schema#11. Note that both issues have been declined in favor of the other. In the meantime, if there is a pressing need, we could try uploading the full complement of Commons sign diagrams to the wiki locally. I’ve just given @matkoniecz some bad memories by floating this idea – it’ll probably just trade one set of problems for another.

If refreshLinks.php is choking on Commons requests, then I wonder to what degree splitting up pages would even help. The script rebuilds the entire il_from table page by page, but my understanding is that MediaWiki’s serial wikitext parser only requests image metadata from the API once per transclusion on a page, not for all the page’s transclusions at once. There would be the same total number of requests. The only fix would be to make the wiki less visual overall, to use fewer distinct images.

Ever since #466 was fixed by installing QuickInstantCommons, I’ve found that the image links are cached very aggressively. If I prospectively embed an image that I only later upload to Commons, it may take days for the page to reflect that upload. I was content with this delay as the tradeoff for being able to load the pages again.

Spitballing a bit: One of the differences between InstantCommons and QuickInstantCommons is that the latter straight-up asks for thumbnails that don’t exist, expecting Commons to generate them on the fly. Do we have pages attempting to create giant thumbnails of JPEGs or complex SVGs? Back when #466 was a noticeable problem, it also affected some short pages that contained only one or two large images, nothing like the pathological sign catalogs I cited.

[matkoniecz]

Would it maybe help to mirror all Wikimedia Commons images used at OSM Wiki to local OSM Wiki? That could be annoying as it would require mirroring also documentation pages (or at least legally required credit part for CC-BY and other attribution-requiring images) and would require monitoring is image deleted on Commons as copyright violation but in principle is doable.

[1ec5]

All of them? That would be around 40,000 distinct images and a lot of disk space. It would be more feasible for the sign catalogs, which generally traffic in relatively simple SVGs in the public domain, but that’s still thousands of images and counting.

This discussion seems like a bit of a catch-all for wiki performance issues attributed to page content. Commons would be one contributing factor, but it isn’t clear that cutting Commons out of the picture would yield a better tradeoff than migrating the Map Features page or one of the other massive pages off the wiki. We could also restart the effort to optimize commonly transcluded templates. I started on more performant rewrites of both {{Tag}} and {{Description}} but got pushback about changing the templates’ behavior. I think those rewrites would be less disruptive than any of the other proposed solutions.

[matkoniecz]

EDIT: This is now posted at https://wiki.openstreetmap.org/wiki/Talk:Wiki#Too_complex_wiki_pages

---

I propose to post at https://wiki.openstreetmap.org/w/index.php?title=Talk:Wiki&action=edit&section=new following

Too complex wiki pages

[[:Category:Pages where template include size is exceeded]] and [[:Category:Pages with too many expensive parser function calls]] are listing some of problematic pages.

This pages are having so many templates that they are causing technical problems - and take loooooooong time to open.

They should be fixed - ideally by somehow improving templates, not using so expensive templates. But splitting them into multiple, removing not needed content, replacing wiki page with some other form of presenting relevant data etc is an option.

See #1046 where this issue was raised. Note that not all problematic pages are found in these categories, but what we have there definitely should be fixed

feel free to propose changes, take it and post on your own or tell me that it is fine

(and yes, I know that one of this pages is mine - I am just testing code change that should limit relevant page size of https://wiki.openstreetmap.org/w/index.php?title=User:Mateusz_Konieczny/unusual_shop_values/United_Kingdom&action=history )

[1ec5]

“OpenHistoricalMap/Projects/Rennes” isn’t a particularly large or complex page, but it took several tries to load the page, each time taking upwards of 3 minutes, due to seven thumbnails of Commons images ranging from 4 to 8 megabytes and one broken image link, which has been fixed. (Subsequent loads came back quickly due to caching.)

[1ec5]

I started on more performant rewrites of both {{Tag}} and {{Description}} but got pushback about changing the templates’ behavior. I think those rewrites would be less disruptive than any of the other proposed solutions.

A Lua-based rewrite of {{tag}} is now ready for review. It isn’t clear to me that this rewrite directly addresses the performance problem analyzed above, but I think it will be a nice incremental improvement for many pages across the site.