Change #1053907 had a related patch set uploaded (by Daniel Kinzler; author: Daniel Kinzler):

[mediawiki/core@master] RefreshLinksJob: collect stats on redundant parses

https://gerrit.wikimedia.org/r/1053907

"if the template is unprotected (protected templates are unlikely to be vandalized)."

But if a protected template is vandalized then the vandalism has a much broader scale.

The English Wikipedia has had LTAs making 500 edits and waiting a month just to vandalize protected templates, for example https://en.wikipedia.org/wiki/Special:Contributions/CheezDeez32. But that tends to already generate complaints about the vandalism hours after it was reverted like https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)/Archive_212#Strange_bug_on_Flag_of_Russia_article even with the recursive purges. But I guess if the initial edit didn't purge things then the revert won't need to either.

when an admin explicitly requests a recursive purge (could be a button on the purge page, or a popup after a revert)

I suspect most vandalism patrollers will have no idea what they are being asked to answer.

I think the idea is probably fine, but you should at least think about how to handle cleanup after cases like the one I linked.

See also:

• (Declined) T250205: Reduce rate of purges emitted by MediaWiki
• (Tracking task) T279205: Harden and improve HTTP cache headers and purging in MediaWiki core (Sprint placeholder)
• Past ideas:
  • "moving dependent purges to RefreshLinksJob"
    • – https://gerrit.wikimedia.org/r/c/mediawiki/core/+/295027
  • https://gerrit.wikimedia.org/r/q/message:RefreshLinksJob+-is:merged

Another option is to subdivide pages into two categories, "high traffic pages" and "long tail low traffic pages". The latter would be put effectively into a no-cache state: the cache lifetime would be very short, and we would never emit purges for them, relying on the natural expiration to deal with vandalism. We'd only emit purges for the high traffic pages.

This is similar to the last item in the "ideas" category except for statically assigning pages to a category (instead of dynamically) and the addition of a no-cache (or limited-lifetime-cache) header for the "low traffic pages". Newly-created pages would probably default to the "high traffic"/"precise purging" category, and we could run a job "every so often" to reasssign the categories, emitting a purge for any page moved from "high traffic" to "low traffic". Nothing needs to be done for a page moved from low traffic to high.

The goal is to increase the amount of /useful work/ done by the purge traffic: increasing the chances that the thing being purged is (a) actually in a cache, and (b) will be viewed from the cache before cache expiration.

The number of resource_change and resource_purge events can get extremely high, spiking at 10k req/sec at times

I'm curious about the the problem that this causes. Too many jobs inserted for job queue to handle quickly enough? Too many purge requests at once?

For completeness, another option is the varnish "x-key" system, which involves two research projects. One is that implementation of x-key in varnish appears to be incomplete, and the second is that the assignment of appropriate x-keys to URLs is non-trivial as well. There are too many templates used on a page like [[Barack Obama]] to naively assign one x-key to every recursively-included template, so we still need to come up with a mechanism to determine which of the templates deserve an x-key assigned, likely based on purge statistics.

Change #1053907 merged by jenkins-bot:

[mediawiki/core@master] RefreshLinksJob: collect stats on redundant parses

https://gerrit.wikimedia.org/r/1053907

In T369898#9990749, @Ottomata wrote:

The number of resource_change and resource_purge events can get extremely high, spiking at 10k req/sec at times

I'm curious about the the problem that this causes. Too many jobs inserted for job queue to handle quickly enough? Too many purge requests at once?

In the case of RESTBase (and the api server cluster) outages or at least incidents. One such incident is documented here https://wikitech.wikimedia.org/wiki/Incidents/2022-03-27_api. It's one of the reasons that I am happy that we are removing RESTBase from the infrastructure.

Looking at metrics from LinksUpdate, it seems that we could reduce the number of purges a lot, if we wait until after we re-parse to decide whetehr we need to purge or not.

On average, we see

• 33 times per second we find a new rendering already cached. This is probably mostly from direct edits. We'd still want to trigger purges on direct edits, immediately.
• 16 times per second, the re-parse generates the exat same HTML as before. We could skip the purge in this case.
• 4 times per seconf we find that the HTML actually changed. So we'd have to purge.
• 61 we don't find anything in the parser cache. There is probably nothing in the edge caches in that case, but we'd still have to notifiy persistant caches (in RESTbase/PCS)

So, of the 114 purges per second, we could:

• skip 16 entirely
• send 61 only to services, not Varnish/ATS