- Training models
- Hausa Wikipedia ha
- Hakka Chinese Wikipedia hak
- Hawaiian Wikipedia haw
- Fiji Hindi Wikipedia hif
- Croatian Wikipedia hr
- Upper Sorbian Wikipedia hsb
- Haitian Creole Wikipedia ht
- ~Armenian Wikipedia hy~ see T308134#8539194
- ~Western Armenian Wikipedia hyw~ see T308134#8548734
- Interlingua Wikipedia ia
- Interlingue Wikipedia ie
- Igbo Wikipedia ig
-
Inupiaq Wikipedia ik - Iloko Wikipedia ilo
- Ingush Wikipedia inh
- Ido Wikipedia io
- Icelandic Wikipedia is
- Inuktitut Wikipedia iu
- Jamaican Patois Wikipedia jam
-
Lojban Wikipedia jbo - Javanese Wikipedia jv
- Models verification
- Publish Datasets
- Populate the excluded section titles
- Deploy back-end
- Check how the model works on the wikis
- In Search, use hasrecommendation:link to find articles
- Test them on https://api.wikimedia.org/service/linkrecommendation/apidocs/#/default/get_v1_linkrecommendations__project___domain___page_title_
- Inform communities
- Deploy front-end
Description
Details
- Due Date
- May 17 2023, 4:00 PM
Status | Subtype | Assigned | Task | ||
---|---|---|---|---|---|
Open | lbowmaker | T307881 Scaling of link suggestions service | |||
Open | Trizek-WMF | T304110 [EPIC] Deploy "add a link" to all Wikipedias | |||
Resolved | Sgs | T308134 Deploy "add a link" to 9th round of wikis | |||
Resolved | None | T327371 Fix Armenian sentence tokenization bug in the link recommendation algorithm |
Event Timeline
20/21 models were trained successfully in the 9th round of wikis.
The Armenian Wikipedia (hywiki) pipeline did not complete successfully and is being investigated in T327371.
Model evaluation has been completed and below are the backtesting results:
Precision@0.5 | Recall@0.5 | |
hawiki | 0.68 | 0.46 |
hakwiki | 0.92 | 0.75 |
hawwiki | 0.92 | 0.74 |
hifwiki | 0.93 | 0.73 |
hrwiki | 0.81 | 0.52 |
hsbwiki | 0.90 | 0.68 |
htwiki | 0.92 | 0.42 |
hywwiki | 0.63 | 0.16 |
iawiki | 0.94 | 0.73 |
iewiki | 0.98 | 0.93 |
igwiki | 0.72 | 0.49 |
ikwiki | 0.92 | 0.60 |
ilowiki | 0.91 | 0.65 |
inhwiki | 0.68 | 0.22 |
iowiki | 0.89 | 0.53 |
iswiki | 0.79 | 0.53 |
iuwiki | 0.96 | 0.72 |
jamwiki | 0.82 | 0.60 |
jbowiki | 0.70 | 0.29 |
jvwiki | 0.87 | 0.61 |
CCing @MGerlach, in case he would like to add comments on the backtesting evaluation.
The conclusion on the backtesting results is that most of the languages look fine besides:
- hywwiki has a low precision (0.63) and recall (0.16).
- igwiki (0.72), jbowiki (0.70), and both hawiki, inhwiki (0.68) have a precision that is slightly lower than the recommended one (0.75).
Talked to @MGerlach about these results and agreed: not to deploy hywwiki; but deploy igwiki, jbowiki, hawiki, and inhwiki since their precision is not too low and the recall is good.
@kostajh, we published datasets for all 19/21 models that passed the evaluation in this round.
I ran this script for adding the link-recommendation task type and and populating the excluded sections:
PHAB=T308134 for WIKI in hawiki hakwiki hawwiki hifwiki hrwiki hsbwiki htwiki hywwiki iawiki iewiki igwiki ikwiki ilowiki inhwiki iowiki iswiki iuwiki jamwiki jbowiki jvwiki; do ORIGIN=`mwscript getConfiguration.php $WIKI --settings 'wgCanonicalServer' --format json | jq --raw-output '.wgCanonicalServer'` mwscript extensions/GrowthExperiments/maintenance/changeWikiConfig.php $WIKI \ --page MediaWiki:NewcomerTasks.json \ --create-only \ --json \ --summary "Growth features configuration boilerplate ([[phab:$PHAB]])" \ link-recommendation \ '{ "type": "link-recommendation", "group": "easy" }' jq "select(.wiki==\"$WIKI\" and .probability > 0.25) | .section" wiki_sections.jsonl \ | jq --slurp --compact-output "unique" \ | mwscript extensions/GrowthExperiments/maintenance/changeWikiConfig.php $WIKI \ --page MediaWiki:NewcomerTasks.json \ --json \ --summary "machine-generated configuration for excluding sections from link recommendations ([[phab:$PHAB]]), feel free to improve" \ link-recommendation.excludedSections \ "`cat`" echo "$ORIGIN/wiki/MediaWiki:NewcomerTasks.json" echo "$ORIGIN/w/index.php?title=MediaWiki:NewcomerTasks.json&diff=next" echo "Press <Enter> to continue" read # give time for manual verification done
I checked the configuration and it seemed to be correctly updated in all wikis. The only mentions worth are ikwiki and ilo which didn't get any excluded section.
Change 892365 had a related patch set uploaded (by Sergio Gimeno; author: Sergio Gimeno):
[operations/mediawiki-config@master] GrowthExperiments: Enable link recommendation for 9th round wikis
Change 892363 had a related patch set uploaded (by Sergio Gimeno; author: Sergio Gimeno):
[operations/mediawiki-config@master] GrowthExperiments: Enable backend of link recommendation for 7,8,9th round wikis
Change 892365 abandoned by Sergio Gimeno:
[operations/mediawiki-config@master] GrowthExperiments: Enable link recommendation for 9th round wikis
Reason:
squashed in I81293b799ec5afe62a19ac2d79e0434047cf1be2
Change 892363 merged by jenkins-bot:
[operations/mediawiki-config@master] GrowthExperiments: Enable backend of link recommendation for 7, 8, 9th round wikis
Mentioned in SAL (#wikimedia-operations) [2023-03-15T20:13:23Z] <samtar@deploy2002> Started scap: Backport for [[gerrit:899673|GrowthExperiments: enable frontend of link recommendation for 6th round wikis (T304550)]], [[gerrit:892363|GrowthExperiments: Enable backend of link recommendation for 7, 8, 9th round wikis (T304551 T308133 T308134)]]
Mentioned in SAL (#wikimedia-operations) [2023-03-15T20:14:55Z] <samtar@deploy2002> sgimeno and samtar: Backport for [[gerrit:899673|GrowthExperiments: enable frontend of link recommendation for 6th round wikis (T304550)]], [[gerrit:892363|GrowthExperiments: Enable backend of link recommendation for 7, 8, 9th round wikis (T304551 T308133 T308134)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet
Mentioned in SAL (#wikimedia-operations) [2023-03-15T20:23:36Z] <samtar@deploy2002> Finished scap: Backport for [[gerrit:899673|GrowthExperiments: enable frontend of link recommendation for 6th round wikis (T304550)]], [[gerrit:892363|GrowthExperiments: Enable backend of link recommendation for 7, 8, 9th round wikis (T304551 T308133 T308134)]] (duration: 10m 12s)
Change 905950 had a related patch set uploaded (by Sergio Gimeno; author: Sergio Gimeno):
[operations/mediawiki-config@master] GrowthExperiments: enable add link frontend and backend
I checked all wikis from this round, this is what I found:
hrwiki and hywiki were correctly enabled but they were missing in the wikis.txt file so not being populated. I've manually added them, cc @kevinbazira
From iewiki to jvwiki were not enabled in 892363, apologies for this, I must have missed them when squashing changes. They will be enabled today at 15h UTC+2.
We should check again all these wikis in a couple of days.
Change 905950 merged by jenkins-bot:
[operations/mediawiki-config@master] GrowthExperiments: enable add link backend in wiki rounds (8,9th)
Mentioned in SAL (#wikimedia-operations) [2023-04-05T13:08:58Z] <lucaswerkmeister-wmde@deploy2002> Started scap: Backport for [[gerrit:905950|GrowthExperiments: enable add link backend in wiki rounds (8,9th) (T308133 T308134)]]
Mentioned in SAL (#wikimedia-operations) [2023-04-05T13:10:28Z] <lucaswerkmeister-wmde@deploy2002> lucaswerkmeister-wmde and sgimeno: Backport for [[gerrit:905950|GrowthExperiments: enable add link backend in wiki rounds (8,9th) (T308133 T308134)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet
Mentioned in SAL (#wikimedia-operations) [2023-04-05T13:16:58Z] <lucaswerkmeister-wmde@deploy2002> Finished scap: Backport for [[gerrit:905950|GrowthExperiments: enable add link backend in wiki rounds (8,9th) (T308133 T308134)]] (duration: 08m 00s)
I tested the models, and everything works fine, except:
- hr.wp returns "There were no results matching the query."
- ie.wp returns "There were no results matching the query."
- ig.wp returns "There were no results matching the query."
- ik.wp returns "There were no results matching the query."
- ilo.wp returns "There were no results matching the query."
- inh.wp returns "There were no results matching the query."
- io.wp returns "There were no results matching the query."
- is.wp returns "There were no results matching the query."
- iu.wp returns "There were no results matching the query."
- is.wp returns "There were no results matching the query."
- jam.wp returns "There were no results matching the query."
- jbo.wp returns "There were no results matching the query."
- jv.wp returns "There were no results matching the query."
We're still investigating, I've added some more details in T308133#8774966 since it seems the same problem affecting gorwiki could also explain why there aren't results on this batch of wikis.
All wikis have now results except jbowiki where link recommendations have been disabled on site. Also worth mentioning that ikwiki is only producing 5 results, so we might want to check that before enabling the frontend.
We can proceed informing communities for the 9th round (except jbo and ik) and add gorwiki from the 8th if you agree. I'll follow up with the issues in jbowiki and ikwiki.
Let's go then with gor + all round 9 (except jbo and ik). Can we deploy next Wednesday, May 17?
I'm adding it to Tech News.
Change 920722 had a related patch set uploaded (by Sergio Gimeno; author: Sergio Gimeno):
[operations/mediawiki-config@master] GrowthExperiments: enable add link frontend in 9th round wikis
Change 920722 merged by jenkins-bot:
[operations/mediawiki-config@master] GrowthExperiments: enable add link frontend in 9th round wikis
Mentioned in SAL (#wikimedia-operations) [2023-05-17T20:03:10Z] <urbanecm@deploy1002> Started scap: Backport for [[gerrit:920722|GrowthExperiments: enable add link frontend in 9th round wikis (T308134)]]
Mentioned in SAL (#wikimedia-operations) [2023-05-17T20:04:46Z] <urbanecm@deploy1002> sgimeno and urbanecm: Backport for [[gerrit:920722|GrowthExperiments: enable add link frontend in 9th round wikis (T308134)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet
Change 920784 had a related patch set uploaded (by Sergio Gimeno; author: Sergio Gimeno):
[operations/mediawiki-config@master] GrowthExperiments: amend wrong wiki prefix for jbowiki
Mentioned in SAL (#wikimedia-operations) [2023-05-17T20:15:17Z] <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:920722|GrowthExperiments: enable add link frontend in 9th round wikis (T308134)]] (duration: 12m 06s)
Change 920784 merged by jenkins-bot:
[operations/mediawiki-config@master] GrowthExperiments: amend wrong wiki prefix for jbowiki
Mentioned in SAL (#wikimedia-operations) [2023-05-17T20:17:29Z] <urbanecm@deploy1002> Started scap: Backport for [[gerrit:920784|GrowthExperiments: amend wrong wiki prefix for jbowiki (T308134)]], [[gerrit:920732|NewTopicOptOutActiveUsers: Skip bot users etc. (T317375)]], [[gerrit:920386|Enable zebra ab test in hewiki (T335972)]]
Mentioned in SAL (#wikimedia-operations) [2023-05-17T20:19:04Z] <urbanecm@deploy1002> urbanecm and matmarex and ksarabia and sgimeno: Backport for [[gerrit:920784|GrowthExperiments: amend wrong wiki prefix for jbowiki (T308134)]], [[gerrit:920732|NewTopicOptOutActiveUsers: Skip bot users etc. (T317375)]], [[gerrit:920386|Enable zebra ab test in hewiki (T335972)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.
Mentioned in SAL (#wikimedia-operations) [2023-05-17T20:29:06Z] <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:920784|GrowthExperiments: amend wrong wiki prefix for jbowiki (T308134)]], [[gerrit:920732|NewTopicOptOutActiveUsers: Skip bot users etc. (T317375)]], [[gerrit:920386|Enable zebra ab test in hewiki (T335972)]] (duration: 11m 36s)
Checked Hawaiian Wikipedia haw, Icelandic Wikipedia is, Inuktitut Wikipedia iu, and Javanese Wikipedia jv - all works as expected.
Checked those - only jbo.wp returns "There were no results matching the query." The Homepage is enabled on jbo.wp (there are options to enable it in Preferences) - no available Suggested edit tasks are displayed.
ikwiki - ikwiki Special:NewcomerTasksInfo does display 5 link-recommendation tasks available, but they are not present in SE module:
@Trizek-WMF , @Sgs - no follow-ups for jbo.wp and ikwiki ?
@Etonkovidova, we decided to exclude these two wikis earlier in this task: T308134#8841112. The task description should have been more explicit, sorry. These wikis will be treated later when all deployments are done.
As everything works as expected, I'm closing this task.