(Translated by https://www.hiragana.jp/)
⚓ T167973 Move database for wikitech (labswiki) to a main cluster section
Page MenuHomePhabricator
Create Task
Maniphest T167973

Move database for wikitech (labswiki) to a main cluster section
Closed, ResolvedPublic
Actions

Description

Historically wikitech has been deployed as self-contained systems with a local MySQL/MariaDB database server co-located with the MediaWiki software. This was done in an attempt to maintain availability of the technical documentation on wikitech in the event of outages in other parts of the network.

Today, we are monitoring and responding to alerts about staleness on the off-site hosted wikitech-static server. The cloud-services-team is also moving forward with a long term plan to remove LDAP auth and OpenStackManager from wikitech so that it can become a normal SUL wiki. These changes mean that eventually wikitech will be hosted in the common MediaWiki wiki farm (T161859). We will rely on the offsite copy of documentation for severe outage support rather than maintaining a single point of failure machine and hoping that it is unaffected.

Moving the labswiki database from m5 to the s3 slice will benefit us with better maintenance, upgrades, monitoring and high availability. Making this move sooner rather than later will reduce the potential to lag behind in schema updates and cause issues with cross-wiki maintenance like we have seen in the past (e.g. T167961).

This could happen at the same time that other movements are being prepared

After the DC switch progress work: T167973#7232200
Migration and clean up progress work: T167973#7359504

Details

SubjectRepoBranchLines +/-
remove s10 referencesoperations/software/conftoolmaster+0 -14
production-m5.sql: Remove labswiki grantsoperations/puppetproduction+0 -28
dumps-*-m5.sql: Remove labswiki grantsoperations/puppetproduction+0 -6
dbbackups: Remove dump grants from m5 for labswikioperations/puppetproduction+0 -6
sections.yaml: Remove s10 from codfwoperations/puppetproduction+0 -1
valid_section.pp: Remove reference to s10operations/puppetproduction+0 -2
Conftool-sections: farewell s10operations/puppetproduction+0 -1
wmf-config: Wikitech migration from s10 to s6operations/mediawiki-configmaster+9 -10
Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
LSobanski added a comment.Mar 19 2021, 2:05 PM

@Andrew I talked to Nicholas earlier this week and here's what the situation is: in Q4 we will be focusing on MariaDB upgrades that will finally be unblocked so there is limited room for other commitments. As a result, this will likely slip into next FY, but is fairly high on our list of things to do.

• Marostegui added a subscriber: • nskaggs.Apr 30 2021, 12:47 PM

We have created a document to try to come up with a movement plan and see if we can do this during the next DC switchover (T281515). Once we've got the plan and procedure a bit clearer we will post it here.
@Andrew @nskaggs please take a look at the shared doc when you can, as there are some things that would need your input.

Thanks

• Marostegui added a comment.May 5 2021, 11:08 AM

I am working on the migration document, making it a lot more detailed and with actual commands. Once that looks good, I will try the procedure on our testing cluster during the next few days. Once done and polished, I will be posting it here

• Marostegui claimed this task.May 5 2021, 11:08 AM
• Marostegui moved this task from Refine to In progress on the DBA board.
bd808 added a subtask: T284106: Replicate & sanitize wikitech data.Jun 1 2021, 8:51 PM
Andrew mentioned this in T284108: Bring labswiki database tables up to date.Jun 7 2021, 2:21 PM
Andrew closed subtask T282209: Enable connection from labweb1001/labweb1002 to s6 as Resolved.Jun 8 2021, 1:56 PM
• Marostegui closed subtask T282074: Audit labswiki grants as Resolved.Jun 10 2021, 11:37 AM
• Marostegui closed subtask T284106: Replicate & sanitize wikitech data as Resolved.Jun 10 2021, 11:43 AM
• Marostegui added a parent task: T285139: DB maintenance work to do while eqiad is passive (June 2021).Jun 18 2021, 11:11 AM
• Marostegui added a comment.EditedJul 23 2021, 10:00 AM

Current status after the work done this week:

  • wikitech database placed in eqiad s6 hosts
  • Replication is enabled on eqiad s6 master (db1173), with multisource and with GTID disabled.
    • There are two channels: m5 (to db1128, m5 master) and s6 (to codfw master)
      • Replication filter for m5:
# mysql.py -hdb1173 -e "show slave 'm5' status\G" | grep Wild
       Replicate_Wild_Do_Table: labswiki.%
   Replicate_Wild_Ignore_Table:
  • Data has been sanitized on db1115 (s6 sanitarium) and its slaves: clouddb1015, clouddb1019, clouddb1021.
    • Check_private_data shows no data. However, there are still no views or grants assigned to labswiki, as I want to do more testing before sending that to WMCS for the views creations.
  • A quick data check shows that replication is working fine, however, I want to do more testing next week to make sure everything is replicating fine

What's next:

Andrew added a comment.Jul 24 2021, 7:20 PM

This is all looking great -- thanks for working on this!

• Marostegui added a comment.Jul 27 2021, 7:20 AM

After a few days replicating, the following tables have been checked between s6 master (db1173) and m5 master (db1128), no differences found:

revision
text
user
actor
ipblocks
comment
user
watchlist
text
logging
page
revision_actor_temp
revision_comment_temp
slots
archive
• Marostegui added a comment.Jul 27 2021, 8:05 AM

Data privacy has been checked:

  • the scripts didn't find anything
  • I manually checked all the actions from the triggers to make sure:
    • the user table and all the new users got sanitized fine
    • No IPs on recentchanges
    • No ar_comment_i different from 0
    • No afl_ip different from ' ';
• Marostegui added a comment.Jul 27 2021, 8:22 AM

Database and grants in place:

# for i in  clouddb1015:3316 clouddb1019:3316 clouddb1021:3316; do mysql.py -h$i labswiki -e "show databases like 'labswiki_p'";done
+-----------------------+
| Database (labswiki_p) |
+-----------------------+
| labswiki_p            |
+-----------------------+
+-----------------------+
| Database (labswiki_p) |
+-----------------------+
| labswiki_p            |
+-----------------------+
+-----------------------+
| Database (labswiki_p) |
+-----------------------+
| labswiki_p            |
+-----------------------+

# for i in  clouddb1015:3316 clouddb1019:3316 clouddb1021:3316; do mysql.py -h$i labswiki -e "show grants for labsdbuser" | grep labswiki;done
GRANT SELECT, SHOW VIEW ON `labswiki_p`.* TO `labsdbuser`
GRANT SELECT, SHOW VIEW ON `labswiki_p`.* TO `labsdbuser`
GRANT SELECT, SHOW VIEW ON `labswiki_p`.* TO `labsdbuser`
• Marostegui updated the task description. (Show Details)Jul 29 2021, 6:40 AM
gerritbot added a comment.Jul 29 2021, 7:22 AM

Change 708716 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/mediawiki-config@master] wmf-config: Wikitech migration from s10 to s6

https://gerrit.wikimedia.org/r/708716

gerritbot added a project: Patch-For-Review.Jul 29 2021, 7:22 AM
• Marostegui added subscribers: Urbanecm, Ladsgroup, Reedy.Jul 29 2021, 7:24 AM

The MW side of things to be done during the RO time for wikitech would be:

  • Change s6.dblists to add labswiki
  • Change s10.dblists to remove labswiki
  • Change db-eqiad.php to move labswiki from s10 to s6.
  • Change db-codfw.php to move labswiki from s10 to s6

This is an initial patch I have created as it might be easier to follow up there: https://gerrit.wikimedia.org/r/708716
I have added people that kindly helped with wiki migrations in the past like @bd808 @Ladsgroup @Urbanecm and @Reedy to see if the can help with the review and to make sure we're not forgetting anything.

Ladsgroup added a comment.Jul 29 2021, 7:30 AM

I'd be more than happy to help in mw side of things, I think we should mark it RO in mediawiki before the change. It should be rather easy.

• Marostegui added a comment.Jul 29 2021, 7:34 AM

Yeah, the idea is to:

  • Set wikitech as RO
  • Do all the changes
  • Set wikitech back to RW and make sure everything works

We want to do this a couple of days after we are back in eqiad.

gerritbot added a comment.Jul 29 2021, 7:59 AM

Change 708631 had a related patch set uploaded (by RhinosF1; author: RhinosF1):

[operations/puppet@production] Conftool-sections: farewell s10

https://gerrit.wikimedia.org/r/708631

gerritbot added a comment.Jul 29 2021, 9:03 AM

Change 708632 had a related patch set uploaded (by RhinosF1; author: RhinosF1):

[operations/software/conftool@master] remove s10 references

https://gerrit.wikimedia.org/r/708632

• Marostegui added subscribers: taavi, RhinosF1.Jul 30 2021, 11:04 AM

Only one point left from the list: T167973#7232200
Let's discuss the code changes on the patchset themselves, although it looks like we've got them pretty clear at this point, and there are just a few minor things to agree upon.
Thanks @Majavah and @RhinosF1 for the quick reviews!

• Marostegui mentioned this in T288093: Place m5 proxies in codfw and eqiad.Aug 4 2021, 12:35 PM
• Marostegui added a comment.Aug 6 2021, 7:42 AM

I have updated the migration document with the last step before the DC switch.
Once we are ready to configure replication cross DC we need to add the following replication filter on the codfw master:

SET GLOBAL replicate_wild_ignore_table = ‘labswiki.%’;

The idea is to avoid replicating labswiki to codfw, as there's no database/tables there yet.

• Marostegui mentioned this in T288720: Failover m5 master (db1128) to db1132 to upgrade its kernel.Aug 12 2021, 6:14 AM
• Bstorm closed subtask T287442: Create views for labswiki (wikitech) as Resolved.Aug 17 2021, 4:19 PM
bd808 mentioned this in T288233: Completion index fails to build on labswiki: MessageCache.php: Process cache for 'en' should be set by now.Aug 18 2021, 9:11 PM
Urbanecm mentioned this in T290810: Special page reports not updating on Wikitech (2021).Sep 12 2021, 2:04 PM
• Marostegui added a comment.EditedSep 14 2021, 6:01 AM

I have updated the doc to reflect the fact that I will be renaming tables on m5 wikitech once we've done the switch to s6, to avoid any possible split brain. It will always help us identify issues or processes that might be using the "old" wikitech location, as original tables will no longer exist with the normal MW name.

set session sql_log_bin=0; rename table abuse_filter to T167973_abuse_filter;
set session sql_log_bin=0; rename table abuse_filter_action to T167973_abuse_filter_action;
set session sql_log_bin=0; rename table abuse_filter_history to T167973_abuse_filter_history;
set session sql_log_bin=0; rename table abuse_filter_log to T167973_abuse_filter_log;
set session sql_log_bin=0; rename table actor to T167973_actor;
set session sql_log_bin=0; rename table archive to T167973_archive;
set session sql_log_bin=0; rename table babel to T167973_babel;
set session sql_log_bin=0; rename table betafeatures_user_counts to T167973_betafeatures_user_counts;
set session sql_log_bin=0; rename table bv2017_edits to T167973_bv2017_edits;
set session sql_log_bin=0; rename table category to T167973_category;
set session sql_log_bin=0; rename table categorylinks to T167973_categorylinks;
set session sql_log_bin=0; rename table change_tag to T167973_change_tag;
set session sql_log_bin=0; rename table change_tag_def to T167973_change_tag_def;
set session sql_log_bin=0; rename table comment to T167973_comment;
set session sql_log_bin=0; rename table config to T167973_config;
set session sql_log_bin=0; rename table content to T167973_content;
set session sql_log_bin=0; rename table content_models to T167973_content_models;
set session sql_log_bin=0; rename table cu_changes to T167973_cu_changes;
set session sql_log_bin=0; rename table cu_log to T167973_cu_log;
set session sql_log_bin=0; rename table discussiontools_subscription to T167973_discussiontools_subscription;
set session sql_log_bin=0; rename table echo_email_batch to T167973_echo_email_batch;
set session sql_log_bin=0; rename table echo_event to T167973_echo_event;
set session sql_log_bin=0; rename table echo_notification to T167973_echo_notification;
set session sql_log_bin=0; rename table echo_target_page to T167973_echo_target_page;
set session sql_log_bin=0; rename table external_user to T167973_external_user;
set session sql_log_bin=0; rename table externallinks to T167973_externallinks;
set session sql_log_bin=0; rename table filearchive to T167973_filearchive;
set session sql_log_bin=0; rename table geo_tags to T167973_geo_tags;
set session sql_log_bin=0; rename table global_block_whitelist to T167973_global_block_whitelist;
set session sql_log_bin=0; rename table globalblocks to T167973_globalblocks;
set session sql_log_bin=0; rename table globalimagelinks to T167973_globalimagelinks;
set session sql_log_bin=0; rename table historical_thread to T167973_historical_thread;
set session sql_log_bin=0; rename table image to T167973_image;
set session sql_log_bin=0; rename table imagelinks to T167973_imagelinks;
set session sql_log_bin=0; rename table interwiki to T167973_interwiki;
set session sql_log_bin=0; rename table ip_changes to T167973_ip_changes;
set session sql_log_bin=0; rename table ipblocks to T167973_ipblocks;
set session sql_log_bin=0; rename table ipblocks_restrictions to T167973_ipblocks_restrictions;
set session sql_log_bin=0; rename table iwlinks to T167973_iwlinks;
set session sql_log_bin=0; rename table job to T167973_job;
set session sql_log_bin=0; rename table l10n_cache to T167973_l10n_cache;
set session sql_log_bin=0; rename table langlinks to T167973_langlinks;
set session sql_log_bin=0; rename table ldap_domains to T167973_ldap_domains;
set session sql_log_bin=0; rename table linter to T167973_linter;
set session sql_log_bin=0; rename table log_search to T167973_log_search;
set session sql_log_bin=0; rename table logging to T167973_logging;
set session sql_log_bin=0; rename table mathoid to T167973_mathoid;
set session sql_log_bin=0; rename table module_deps to T167973_module_deps;
set session sql_log_bin=0; rename table oathauth_users to T167973_oathauth_users;
set session sql_log_bin=0; rename table oathauth_users_restore to T167973_oathauth_users_restore;
set session sql_log_bin=0; rename table oauth2_access_tokens to T167973_oauth2_access_tokens;
set session sql_log_bin=0; rename table oauth_accepted_consumer to T167973_oauth_accepted_consumer;
set session sql_log_bin=0; rename table oauth_registered_consumer to T167973_oauth_registered_consumer;
set session sql_log_bin=0; rename table objectcache to T167973_objectcache;
set session sql_log_bin=0; rename table oldimage to T167973_oldimage;
set session sql_log_bin=0; rename table page to T167973_page;
set session sql_log_bin=0; rename table page_props to T167973_page_props;
set session sql_log_bin=0; rename table page_restrictions to T167973_page_restrictions;
set session sql_log_bin=0; rename table pagelinks to T167973_pagelinks;
set session sql_log_bin=0; rename table protected_titles to T167973_protected_titles;
set session sql_log_bin=0; rename table querycache to T167973_querycache;
set session sql_log_bin=0; rename table querycache_info to T167973_querycache_info;
set session sql_log_bin=0; rename table querycachetwo to T167973_querycachetwo;
set session sql_log_bin=0; rename table recentchanges to T167973_recentchanges;
set session sql_log_bin=0; rename table redirect to T167973_redirect;
set session sql_log_bin=0; rename table revision to T167973_revision;
set session sql_log_bin=0; rename table revision_actor_temp to T167973_revision_actor_temp;
set session sql_log_bin=0; rename table revision_comment_temp to T167973_revision_comment_temp;
set session sql_log_bin=0; rename table searchindex to T167973_searchindex;
set session sql_log_bin=0; rename table site_identifiers to T167973_site_identifiers;
set session sql_log_bin=0; rename table site_stats to T167973_site_stats;
set session sql_log_bin=0; rename table sites to T167973_sites;
set session sql_log_bin=0; rename table slot_roles to T167973_slot_roles;
set session sql_log_bin=0; rename table slots to T167973_slots;
set session sql_log_bin=0; rename table spoofuser to T167973_spoofuser;
set session sql_log_bin=0; rename table templatelinks to T167973_templatelinks;
set session sql_log_bin=0; rename table text to T167973_text;
set session sql_log_bin=0; rename table thread to T167973_thread;
set session sql_log_bin=0; rename table thread_history to T167973_thread_history;
set session sql_log_bin=0; rename table thread_pending_relationship to T167973_thread_pending_relationship;
set session sql_log_bin=0; rename table thread_reaction to T167973_thread_reaction;
set session sql_log_bin=0; rename table updatelog to T167973_updatelog;
set session sql_log_bin=0; rename table uploadstash to T167973_uploadstash;
set session sql_log_bin=0; rename table user to T167973_user;
set session sql_log_bin=0; rename table user_former_groups to T167973_user_former_groups;
set session sql_log_bin=0; rename table user_groups to T167973_user_groups;
set session sql_log_bin=0; rename table user_message_state to T167973_user_message_state;
set session sql_log_bin=0; rename table user_newtalk to T167973_user_newtalk;
set session sql_log_bin=0; rename table user_properties to T167973_user_properties;
set session sql_log_bin=0; rename table watchlist to T167973_watchlist;
set session sql_log_bin=0; rename table watchlist_expiry to T167973_watchlist_expiry;
set session sql_log_bin=0; rename table wikilove_log to T167973_wikilove_log;

To roll that back:

set session sql_log_bin=0;rename table T167973_abuse_filter to abuse_filter;
set session sql_log_bin=0;rename table T167973_abuse_filter_action to abuse_filter_action;
set session sql_log_bin=0;rename table T167973_abuse_filter_history to abuse_filter_history;
set session sql_log_bin=0;rename table T167973_abuse_filter_log to abuse_filter_log;
set session sql_log_bin=0;rename table T167973_actor to actor;
set session sql_log_bin=0;rename table T167973_archive to archive;
set session sql_log_bin=0;rename table T167973_babel to babel;
set session sql_log_bin=0;rename table T167973_betafeatures_user_counts to betafeatures_user_counts;
set session sql_log_bin=0;rename table T167973_bv2017_edits to bv2017_edits;
set session sql_log_bin=0;rename table T167973_category to category;
set session sql_log_bin=0;rename table T167973_categorylinks to categorylinks;
set session sql_log_bin=0;rename table T167973_change_tag to change_tag;
set session sql_log_bin=0;rename table T167973_change_tag_def to change_tag_def;
set session sql_log_bin=0;rename table T167973_comment to comment;
set session sql_log_bin=0;rename table T167973_config to config;
set session sql_log_bin=0;rename table T167973_content to content;
set session sql_log_bin=0;rename table T167973_content_models to content_models;
set session sql_log_bin=0;rename table T167973_cu_changes to cu_changes;
set session sql_log_bin=0;rename table T167973_cu_log to cu_log;
set session sql_log_bin=0;rename table T167973_discussiontools_subscription to discussiontools_subscription;
set session sql_log_bin=0;rename table T167973_echo_email_batch to echo_email_batch;
set session sql_log_bin=0;rename table T167973_echo_event to echo_event;
set session sql_log_bin=0;rename table T167973_echo_notification to echo_notification;
set session sql_log_bin=0;rename table T167973_echo_target_page to echo_target_page;
set session sql_log_bin=0;rename table T167973_external_user to external_user;
set session sql_log_bin=0;rename table T167973_externallinks to externallinks;
set session sql_log_bin=0;rename table T167973_filearchive to filearchive;
set session sql_log_bin=0;rename table T167973_geo_tags to geo_tags;
set session sql_log_bin=0;rename table T167973_global_block_whitelist to global_block_whitelist;
set session sql_log_bin=0;rename table T167973_globalblocks to globalblocks;
set session sql_log_bin=0;rename table T167973_globalimagelinks to globalimagelinks;
set session sql_log_bin=0;rename table T167973_historical_thread to historical_thread;
set session sql_log_bin=0;rename table T167973_image to image;
set session sql_log_bin=0;rename table T167973_imagelinks to imagelinks;
set session sql_log_bin=0;rename table T167973_interwiki to interwiki;
set session sql_log_bin=0;rename table T167973_ip_changes to ip_changes;
set session sql_log_bin=0;rename table T167973_ipblocks to ipblocks;
set session sql_log_bin=0;rename table T167973_ipblocks_restrictions to ipblocks_restrictions;
set session sql_log_bin=0;rename table T167973_iwlinks to iwlinks;
set session sql_log_bin=0;rename table T167973_job to job;
set session sql_log_bin=0;rename table T167973_l10n_cache to l10n_cache;
set session sql_log_bin=0;rename table T167973_langlinks to langlinks;
set session sql_log_bin=0;rename table T167973_ldap_domains to ldap_domains;
set session sql_log_bin=0;rename table T167973_linter to linter;
set session sql_log_bin=0;rename table T167973_log_search to log_search;
set session sql_log_bin=0;rename table T167973_logging to logging;
set session sql_log_bin=0;rename table T167973_mathoid to mathoid;
set session sql_log_bin=0;rename table T167973_module_deps to module_deps;
set session sql_log_bin=0;rename table T167973_oathauth_users to oathauth_users;
set session sql_log_bin=0;rename table T167973_oathauth_users_restore to oathauth_users_restore;
set session sql_log_bin=0;rename table T167973_oauth2_access_tokens to oauth2_access_tokens;
set session sql_log_bin=0;rename table T167973_oauth_accepted_consumer to oauth_accepted_consumer;
set session sql_log_bin=0;rename table T167973_oauth_registered_consumer to oauth_registered_consumer;
set session sql_log_bin=0;rename table T167973_objectcache to objectcache;
set session sql_log_bin=0;rename table T167973_oldimage to oldimage;
set session sql_log_bin=0;rename table T167973_page to page;
set session sql_log_bin=0;rename table T167973_page_props to page_props;
set session sql_log_bin=0;rename table T167973_page_restrictions to page_restrictions;
set session sql_log_bin=0;rename table T167973_pagelinks to pagelinks;
set session sql_log_bin=0;rename table T167973_protected_titles to protected_titles;
set session sql_log_bin=0;rename table T167973_querycache to querycache;
set session sql_log_bin=0;rename table T167973_querycache_info to querycache_info;
set session sql_log_bin=0;rename table T167973_querycachetwo to querycachetwo;
set session sql_log_bin=0;rename table T167973_recentchanges to recentchanges;
set session sql_log_bin=0;rename table T167973_redirect to redirect;
set session sql_log_bin=0;rename table T167973_revision to revision;
set session sql_log_bin=0;rename table T167973_revision_actor_temp to revision_actor_temp;
set session sql_log_bin=0;rename table T167973_revision_comment_temp to revision_comment_temp;
set session sql_log_bin=0;rename table T167973_searchindex to searchindex;
set session sql_log_bin=0;rename table T167973_site_identifiers to site_identifiers;
set session sql_log_bin=0;rename table T167973_site_stats to site_stats;
set session sql_log_bin=0;rename table T167973_sites to sites;
set session sql_log_bin=0;rename table T167973_slot_roles to slot_roles;
set session sql_log_bin=0;rename table T167973_slots to slots;
set session sql_log_bin=0;rename table T167973_spoofuser to spoofuser;
set session sql_log_bin=0;rename table T167973_templatelinks to templatelinks;
set session sql_log_bin=0;rename table T167973_text to text;
set session sql_log_bin=0;rename table T167973_thread to thread;
set session sql_log_bin=0;rename table T167973_thread_history to thread_history;
set session sql_log_bin=0;rename table T167973_thread_pending_relationship to thread_pending_relationship;
set session sql_log_bin=0;rename table T167973_thread_reaction to thread_reaction;
set session sql_log_bin=0;rename table T167973_updatelog to updatelog;
set session sql_log_bin=0;rename table T167973_uploadstash to uploadstash;
set session sql_log_bin=0;rename table T167973_user to user;
set session sql_log_bin=0;rename table T167973_user_former_groups to user_former_groups;
set session sql_log_bin=0;rename table T167973_user_groups to user_groups;
set session sql_log_bin=0;rename table T167973_user_message_state to user_message_state;
set session sql_log_bin=0;rename table T167973_user_newtalk to user_newtalk;
set session sql_log_bin=0;rename table T167973_user_properties to user_properties;
set session sql_log_bin=0;rename table T167973_watchlist to watchlist;
set session sql_log_bin=0;rename table T167973_watchlist_expiry to watchlist_expiry;
set session sql_log_bin=0;rename table T167973_wikilove_log to wikilove_log;
• Marostegui added a comment.Sep 14 2021, 6:43 AM

Pending patches to be merged during/after the migration:

• Marostegui added a comment.Sep 15 2021, 4:50 AM

This is scheduled for Thursday at 15:00 UTC

Stashbot added a comment.Sep 16 2021, 3:03 PM

Mentioned in SAL (#wikimedia-operations) [2021-09-16T15:03:24Z] <marostegui> Set wikitech on read-only (from now on all SAL changes will fail) T167973

gerritbot added a comment.Sep 16 2021, 3:09 PM

Change 708716 merged by jenkins-bot:

[operations/mediawiki-config@master] wmf-config: Wikitech migration from s10 to s6

https://gerrit.wikimedia.org/r/708716

Stashbot added a comment.Sep 16 2021, 3:19 PM

Mentioned in SAL (#wikimedia-operations) [2021-09-16T15:19:24Z] <marostegui@deploy1002> Synchronized wmf-config/db-codfw.php: Wikitech move from s10 to s6 T167973 (duration: 01m 05s)

Stashbot added a comment.Sep 16 2021, 3:36 PM

Mentioned in SAL (#wikimedia-operations) [2021-09-16T15:36:21Z] <marostegui@deploy1002> Synchronized wmf-config/db-eqiad.php: Wikitech move from s10 to s6 T167973 (duration: 01m 05s)

Stashbot added a comment.Sep 16 2021, 3:38 PM

Mentioned in SAL (#wikimedia-operations) [2021-09-16T15:38:18Z] <marostegui@deploy1002> Synchronized dblists/s6.dblist: Wikitech move from s10 to s6 T167973 (duration: 01m 05s)

Stashbot added a comment.Sep 16 2021, 3:39 PM

Mentioned in SAL (#wikimedia-operations) [2021-09-16T15:39:48Z] <marostegui@deploy1002> Synchronized dblists/: Wikitech move from s10 to s6 T167973 (duration: 01m 05s)

Stashbot added a comment.Sep 16 2021, 3:41 PM

Mentioned in SAL (#wikimedia-operations) [2021-09-16T15:41:02Z] <marostegui@deploy1002> Synchronized tests/dblistTest.php: Wikitech move from s10 to s6 T167973 (duration: 01m 05s)

Stashbot added a comment.Sep 16 2021, 3:42 PM

Mentioned in SAL (#wikimedia-operations) [2021-09-16T15:42:15Z] <marostegui@deploy1002> Synchronized wmf-config/config/labswiki.yaml: Wikitech move from s10 to s6 T167973 (duration: 01m 04s)

bd808 awarded a token.Sep 16 2021, 3:55 PM
• Marostegui added a subscriber: Krinkle.EditedSep 16 2021, 4:02 PM

This has been done today. It has taken a lot of effort and support from lots of people, and a lot more people than expected showed up on IRC to help! Much much appreciated.
Thank you: @jcrespo @Reedy @Krinkle @Ladsgroup @Urbanecm @bd808 @Andrew moving this was very complex given how of a snowflake wikitech is (was!).

In general the procedure was a bit chaotic, especially on the order of how the files needed to be synced, the order used was this in the end:

  • wmf-config/db-codfw.php
  • wmf-config/db-eqiad.php
  • dblists/s6.dblist
  • dblists/
  • tests/dblistTest.php
  • wmf-config/config/labswiki.yaml

We also found that labweb1001 and labweb1002 grants for heartbeat were missing (as they are not in m5, they were not migrated), so they needed to be added:

GRANT SELECT ON `heartbeat`.`heartbeat` TO `wikiuser`@`208.80.155.109`
GRANT SELECT ON `heartbeat`.`heartbeat` TO `wikiuser`@`208.80.154.160`
GRANT SELECT ON `heartbeat`.`heartbeat` TO `wikiadmin`@`208.80.154.160`;
GRANT SELECT ON `heartbeat`.`heartbeat` TO `wikiadmin`@`208.80.155.109`;

Tables on old m5 masters are still there but renamed, with this pattern: T167973#7350916
T167973_NAME:

If something tries to write to them, it will fail.

What's next in terms of finishing the whole task:

Ladsgroup awarded a token.Sep 16 2021, 4:04 PM
Stashbot added a comment.Sep 16 2021, 4:04 PM

Mentioned in SAL (#wikimedia-operations) [2021-09-16T16:04:13Z] <marostegui> Disconnect s6 master from m5 master (noting the replication position) T167973

• Marostegui added a comment.Sep 16 2021, 4:04 PM

s6 channel replicating from m5 file and position:

root@db1173.eqiad.wmnet[(none)]> show all slaves status\G
*************************** 1. row ***************************
               Connection_name: m5
               Slave_SQL_State:
                Slave_IO_State:
                   Master_Host: db1128.eqiad.wmnet
                   Master_User: repl
                   Master_Port: 3306
                 Connect_Retry: 60
               Master_Log_File: db1128-bin.000250
           Read_Master_Log_Pos: 390892003
                Relay_Log_File: db1173-relay-bin-m5.000013
                 Relay_Log_Pos: 390892303
         Relay_Master_Log_File: db1128-bin.000250
              Slave_IO_Running: No
             Slave_SQL_Running: No
               Replicate_Do_DB:
           Replicate_Ignore_DB:
            Replicate_Do_Table:
        Replicate_Ignore_Table:
       Replicate_Wild_Do_Table: labswiki.%
   Replicate_Wild_Ignore_Table:
                    Last_Errno: 0
                    Last_Error:
                  Skip_Counter: 0
           Exec_Master_Log_Pos: 390892003
               Relay_Log_Space: 390892664
               Until_Condition: None
                Until_Log_File:
                 Until_Log_Pos: 0
            Master_SSL_Allowed: Yes
            Master_SSL_CA_File:
            Master_SSL_CA_Path:
               Master_SSL_Cert:
             Master_SSL_Cipher:
                Master_SSL_Key:
         Seconds_Behind_Master: NULL
 Master_SSL_Verify_Server_Cert: No
                 Last_IO_Errno: 0
                 Last_IO_Error:
                Last_SQL_Errno: 0
                Last_SQL_Error:
   Replicate_Ignore_Server_Ids:
              Master_Server_Id: 171966562
                Master_SSL_Crl:
            Master_SSL_Crlpath:
                    Using_Gtid: No
                   Gtid_IO_Pos:
       Replicate_Do_Domain_Ids:
   Replicate_Ignore_Domain_Ids:
                 Parallel_Mode: conservative
                     SQL_Delay: 0
           SQL_Remaining_Delay: NULL
       Slave_SQL_Running_State:
              Slave_DDL_Groups: 36
Slave_Non_Transactional_Groups: 0
    Slave_Transactional_Groups: 5937038
          Retried_transactions: 0
            Max_relay_log_size: 1048576000
          Executed_log_entries: 20336797
     Slave_received_heartbeats: 0
        Slave_heartbeat_period: 30.000
                Gtid_Slave_Pos: 0-180359184-3049354376,171966562-171966562-80913712,171970594-171970594-1063329989,171970705-171970705-239075865,171974662-171974662-836768948,171974728-171974728-338486249,171974883-171974883-1921892293,171978766-171978766-1375989821,171978805-171978805-185121095,171978904-171978904-198543478,180359184-180359184-35598956,180363370-180363370-7087924,180367474-180367474-91976046,180367475-180367475-610349294
• Marostegui mentioned this in T291144: 2021 codfw -> eqiad switch post DB work.Sep 16 2021, 4:12 PM
jcrespo awarded a token.Sep 16 2021, 4:44 PM
• Marostegui added a comment.Sep 20 2021, 5:00 AM

No errors reported on labswiki since the migration, so this is looking good.

Stashbot added a comment.Sep 20 2021, 7:31 AM

Mentioned in SAL (#wikimedia-operations) [2021-09-20T07:31:42Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1096:3316 T167973', diff saved to https://phabricator.wikimedia.org/P17297 and previous config saved to /var/cache/conftool/dbconfig/20210920-073141-marostegui.json

Stashbot added a comment.Sep 20 2021, 7:32 AM

Mentioned in SAL (#wikimedia-operations) [2021-09-20T07:32:07Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1096:3316 T167973', diff saved to https://phabricator.wikimedia.org/P17298 and previous config saved to /var/cache/conftool/dbconfig/20210920-073206-marostegui.json

Stashbot added a comment.Sep 20 2021, 7:32 AM

Mentioned in SAL (#wikimedia-operations) [2021-09-20T07:32:56Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1168 T167973', diff saved to https://phabricator.wikimedia.org/P17299 and previous config saved to /var/cache/conftool/dbconfig/20210920-073256-marostegui.json

Stashbot added a comment.Sep 20 2021, 7:35 AM

Mentioned in SAL (#wikimedia-operations) [2021-09-20T07:35:58Z] <marostegui> Stop db1168 and db2129 in sync T167973

gerritbot added a comment.Sep 20 2021, 9:33 AM

Change 708631 merged by Marostegui:

[operations/puppet@production] Conftool-sections: farewell s10

https://gerrit.wikimedia.org/r/708631

gerritbot added a comment.Sep 20 2021, 9:35 AM

Change 722273 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] sections.yaml: Remove s10 from codfw

https://gerrit.wikimedia.org/r/722273

Stashbot added a comment.Sep 20 2021, 9:47 AM

Mentioned in SAL (#wikimedia-operations) [2021-09-20T09:47:39Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Remove s10 from eqiad T167973', diff saved to https://phabricator.wikimedia.org/P17300 and previous config saved to /var/cache/conftool/dbconfig/20210920-094739-marostegui.json

• Marostegui added a comment.Sep 20 2021, 10:57 AM

I have placed labswiki now in codfw too. I have compared the data and there are no differences and new changes are happening just fine in codfw too.

• Marostegui added a comment.Sep 21 2021, 5:18 AM

s6 eqiad dump contains labswiki already:

root@dbprov1003:/srv/backups/dumps/latest/dump.s6.2021-09-21--03-20-38# ls -lh | grep labswiki
-rw-r--r-- 1 dump dump   271 Sep 21 04:13 labswiki.abuse_filter_action-schema.sql.gz
-rw-r--r-- 1 dump dump   243 Sep 21 03:45 labswiki.abuse_filter_action.sql.gz
-rw-r--r-- 1 dump dump   408 Sep 21 04:13 labswiki.abuse_filter_history-schema.sql.gz
-rw-r--r-- 1 dump dump  2.3K Sep 21 03:45 labswiki.abuse_filter_history.sql.gz
-rw-r--r-- 1 dump dump   483 Sep 21 04:13 labswiki.abuse_filter_log-schema.sql.gz
-rw-r--r-- 1 dump dump  7.3K Sep 21 03:45 labswiki.abuse_filter_log.sql.gz
-rw-r--r-- 1 dump dump   421 Sep 21 04:13 labswiki.abuse_filter-schema.sql.gz
-rw-r--r-- 1 dump dump  1.5K Sep 21 03:45 labswiki.abuse_filter.sql.gz
-rw-r--r-- 1 dump dump   280 Sep 21 04:13 labswiki.actor-schema.sql.gz
-rw-r--r-- 1 dump dump  332K Sep 21 03:45 labswiki.actor.sql.gz
-rw-r--r-- 1 dump dump   424 Sep 21 04:13 labswiki.archive-schema.sql.gz
-rw-r--r-- 1 dump dump   55M Sep 21 03:45 labswiki.archive.sql.gz
-rw-r--r-- 1 dump dump   250 Sep 21 04:13 labswiki.babel-schema.sql.gz
-rw-r--r-- 1 dump dump  2.2K Sep 21 03:45 labswiki.babel.sql.gz
-rw-r--r-- 1 dump dump   232 Sep 21 04:13 labswiki.betafeatures_user_counts-schema.sql.gz
-rw-r--r-- 1 dump dump   240 Sep 21 03:45 labswiki.betafeatures_user_counts.sql.gz
-rw-r--r-- 1 dump dump   223 Sep 21 04:13 labswiki.bv2017_edits-schema.sql.gz
-rw-r--r-- 1 dump dump  8.0K Sep 21 03:45 labswiki.bv2017_edits.sql.gz
-rw-r--r-- 1 dump dump   378 Sep 21 04:13 labswiki.categorylinks-schema.sql.gz
-rw-r--r-- 1 dump dump  228K Sep 21 03:45 labswiki.categorylinks.sql.gz
-rw-r--r-- 1 dump dump   295 Sep 21 04:13 labswiki.category-schema.sql.gz
-rw-r--r-- 1 dump dump  5.9K Sep 21 03:45 labswiki.category.sql.gz
-rw-r--r-- 1 dump dump   312 Sep 21 04:13 labswiki.change_tag_def-schema.sql.gz
-rw-r--r-- 1 dump dump   499 Sep 21 03:45 labswiki.change_tag_def.sql.gz
-rw-r--r-- 1 dump dump   322 Sep 21 04:13 labswiki.change_tag-schema.sql.gz
-rw-r--r-- 1 dump dump  137K Sep 21 03:45 labswiki.change_tag.sql.gz
-rw-r--r-- 1 dump dump   274 Sep 21 04:13 labswiki.comment-schema.sql.gz
-rw-r--r-- 1 dump dump  8.0M Sep 21 03:45 labswiki.comment.sql.gz
-rw-r--r-- 1 dump dump   232 Sep 21 04:13 labswiki.config-schema.sql.gz
-rw-r--r-- 1 dump dump   251 Sep 21 04:13 labswiki.content_models-schema.sql.gz
-rw-r--r-- 1 dump dump   240 Sep 21 03:45 labswiki.content_models.sql.gz
-rw-r--r-- 1 dump dump   286 Sep 21 04:13 labswiki.content-schema.sql.gz
-rw-r--r-- 1 dump dump   49M Sep 21 03:45 labswiki.content.sql.gz
-rw-r--r-- 1 dump dump   457 Sep 21 04:13 labswiki.cu_changes-schema.sql.gz
-rw-r--r-- 1 dump dump  1.1M Sep 21 03:45 labswiki.cu_changes.sql.gz
-rw-r--r-- 1 dump dump   378 Sep 21 04:13 labswiki.cu_log-schema.sql.gz
-rw-r--r-- 1 dump dump  4.4K Sep 21 03:45 labswiki.cu_log.sql.gz
-rw-r--r-- 1 dump dump   348 Sep 21 04:13 labswiki.discussiontools_subscription-schema.sql.gz
-rw-r--r-- 1 dump dump   327 Sep 21 04:13 labswiki.echo_email_batch-schema.sql.gz
-rw-r--r-- 1 dump dump   604 Sep 21 03:45 labswiki.echo_email_batch.sql.gz
-rw-r--r-- 1 dump dump   347 Sep 21 04:13 labswiki.echo_event-schema.sql.gz
-rw-r--r-- 1 dump dump  919K Sep 21 03:45 labswiki.echo_event.sql.gz
-rw-r--r-- 1 dump dump   369 Sep 21 04:13 labswiki.echo_notification-schema.sql.gz
-rw-r--r-- 1 dump dump  1.2M Sep 21 03:45 labswiki.echo_notification.sql.gz
-rw-r--r-- 1 dump dump   293 Sep 21 04:13 labswiki.echo_target_page-schema.sql.gz
-rw-r--r-- 1 dump dump  1.1M Sep 21 03:45 labswiki.echo_target_page.sql.gz
-rw-r--r-- 1 dump dump   325 Sep 21 04:13 labswiki.externallinks-schema.sql.gz
-rw-r--r-- 1 dump dump  962K Sep 21 03:45 labswiki.externallinks.sql.gz
-rw-r--r-- 1 dump dump   248 Sep 21 04:13 labswiki.external_user-schema.sql.gz
-rw-r--r-- 1 dump dump   639 Sep 21 04:13 labswiki.filearchive-schema.sql.gz
-rw-r--r-- 1 dump dump  9.3K Sep 21 03:45 labswiki.filearchive.sql.gz
-rw-r--r-- 1 dump dump   339 Sep 21 04:13 labswiki.geo_tags-schema.sql.gz
-rw-r--r-- 1 dump dump   370 Sep 21 04:13 labswiki.globalblocks-schema.sql.gz
-rw-r--r-- 1 dump dump   280 Sep 21 04:13 labswiki.global_block_whitelist-schema.sql.gz
-rw-r--r-- 1 dump dump   307 Sep 21 04:13 labswiki.globalimagelinks-schema.sql.gz
-rw-r--r-- 1 dump dump   272 Sep 21 04:13 labswiki.historical_thread-schema.sql.gz
-rw-r--r-- 1 dump dump   281 Sep 21 04:13 labswiki.imagelinks-schema.sql.gz
-rw-r--r-- 1 dump dump   47K Sep 21 03:45 labswiki.imagelinks.sql.gz
-rw-r--r-- 1 dump dump   547 Sep 21 04:13 labswiki.image-schema.sql.gz
-rw-r--r-- 1 dump dump  316K Sep 21 03:45 labswiki.image.sql.gz
-rw-r--r-- 1 dump dump   263 Sep 21 04:13 labswiki.interwiki-schema.sql.gz
-rw-r--r-- 1 dump dump   251 Sep 21 04:13 labswiki.ipblocks_restrictions-schema.sql.gz
-rw-r--r-- 1 dump dump   502 Sep 21 04:13 labswiki.ipblocks-schema.sql.gz
-rw-r--r-- 1 dump dump   77K Sep 21 03:45 labswiki.ipblocks.sql.gz
-rw-r--r-- 1 dump dump   273 Sep 21 04:13 labswiki.ip_changes-schema.sql.gz
-rw-r--r-- 1 dump dump   20K Sep 21 03:45 labswiki.ip_changes.sql.gz
-rw-r--r-- 1 dump dump   280 Sep 21 04:13 labswiki.iwlinks-schema.sql.gz
-rw-r--r-- 1 dump dump  208K Sep 21 03:45 labswiki.iwlinks.sql.gz
-rw-r--r-- 1 dump dump   402 Sep 21 04:13 labswiki.job-schema.sql.gz
-rw-r--r-- 1 dump dump  840K Sep 21 03:45 labswiki.job.sql.gz
-rw-r--r-- 1 dump dump   238 Sep 21 04:13 labswiki.l10n_cache-schema.sql.gz
-rw-r--r-- 1 dump dump   260 Sep 21 04:13 labswiki.langlinks-schema.sql.gz
-rw-r--r-- 1 dump dump  3.9K Sep 21 03:45 labswiki.langlinks.sql.gz
-rw-r--r-- 1 dump dump   256 Sep 21 04:13 labswiki.ldap_domains-schema.sql.gz
-rw-r--r-- 1 dump dump  271K Sep 21 03:45 labswiki.ldap_domains.sql.gz
-rw-r--r-- 1 dump dump   305 Sep 21 04:13 labswiki.linter-schema.sql.gz
-rw-r--r-- 1 dump dump   32K Sep 21 03:45 labswiki.linter.sql.gz
-rw-r--r-- 1 dump dump   440 Sep 21 04:13 labswiki.logging-schema.sql.gz
-rw-r--r-- 1 dump dump  6.5M Sep 21 03:45 labswiki.logging.sql.gz
-rw-r--r-- 1 dump dump   258 Sep 21 04:13 labswiki.log_search-schema.sql.gz
-rw-r--r-- 1 dump dump  123K Sep 21 03:45 labswiki.log_search.sql.gz
-rw-r--r-- 1 dump dump   271 Sep 21 04:13 labswiki.mathoid-schema.sql.gz
-rw-r--r-- 1 dump dump   233 Sep 21 04:13 labswiki.module_deps-schema.sql.gz
-rw-r--r-- 1 dump dump   81K Sep 21 03:45 labswiki.module_deps.sql.gz
-rw-r--r-- 1 dump dump   276 Sep 21 04:13 labswiki.oathauth_users_restore-schema.sql.gz
-rw-r--r-- 1 dump dump   17K Sep 21 03:45 labswiki.oathauth_users_restore.sql.gz
-rw-r--r-- 1 dump dump   231 Sep 21 04:13 labswiki.oathauth_users-schema.sql.gz
-rw-r--r-- 1 dump dump   81K Sep 21 03:45 labswiki.oathauth_users.sql.gz
-rw-r--r-- 1 dump dump   314 Sep 21 04:13 labswiki.oauth2_access_tokens-schema.sql.gz
-rw-r--r-- 1 dump dump   386 Sep 21 04:13 labswiki.oauth_accepted_consumer-schema.sql.gz
-rw-r--r-- 1 dump dump  1.7K Sep 21 03:45 labswiki.oauth_accepted_consumer.sql.gz
-rw-r--r-- 1 dump dump   562 Sep 21 04:13 labswiki.oauth_registered_consumer-schema.sql.gz
-rw-r--r-- 1 dump dump  2.0K Sep 21 03:45 labswiki.oauth_registered_consumer.sql.gz
-rw-r--r-- 1 dump dump   256 Sep 21 04:13 labswiki.objectcache-schema.sql.gz
-rw-r--r-- 1 dump dump   21K Sep 21 03:45 labswiki.objectcache.sql.gz
-rw-r--r-- 1 dump dump   553 Sep 21 04:13 labswiki.oldimage-schema.sql.gz
-rw-r--r-- 1 dump dump   30K Sep 21 03:45 labswiki.oldimage.sql.gz
-rw-r--r-- 1 dump dump   290 Sep 21 04:13 labswiki.pagelinks-schema.sql.gz
-rw-r--r-- 1 dump dump  261K Sep 21 03:45 labswiki.pagelinks.sql.gz
-rw-r--r-- 1 dump dump   278 Sep 21 04:13 labswiki.page_props-schema.sql.gz
-rw-r--r-- 1 dump dump   58K Sep 21 03:45 labswiki.page_props.sql.gz
-rw-r--r-- 1 dump dump   341 Sep 21 04:13 labswiki.page_restrictions-schema.sql.gz
-rw-r--r-- 1 dump dump  1.8K Sep 21 03:45 labswiki.page_restrictions.sql.gz
-rw-r--r-- 1 dump dump   434 Sep 21 04:13 labswiki.page-schema.sql.gz
-rw-r--r-- 1 dump dump  744K Sep 21 03:45 labswiki.page.sql.gz
-rw-r--r-- 1 dump dump   310 Sep 21 04:13 labswiki.protected_titles-schema.sql.gz
-rw-r--r-- 1 dump dump   291 Sep 21 03:45 labswiki.protected_titles.sql.gz
-rw-r--r-- 1 dump dump   243 Sep 21 04:13 labswiki.querycache_info-schema.sql.gz
-rw-r--r-- 1 dump dump   461 Sep 21 03:45 labswiki.querycache_info.sql.gz
-rw-r--r-- 1 dump dump   260 Sep 21 04:13 labswiki.querycache-schema.sql.gz
-rw-r--r-- 1 dump dump  449K Sep 21 03:45 labswiki.querycache.sql.gz
-rw-r--r-- 1 dump dump   298 Sep 21 04:13 labswiki.querycachetwo-schema.sql.gz
-rw-r--r-- 1 dump dump  1.6K Sep 21 03:45 labswiki.querycachetwo.sql.gz
-rw-r--r-- 1 dump dump   537 Sep 21 04:13 labswiki.recentchanges-schema.sql.gz
-rw-r--r-- 1 dump dump  116K Sep 21 03:45 labswiki.recentchanges.sql.gz
-rw-r--r-- 1 dump dump   293 Sep 21 04:13 labswiki.redirect-schema.sql.gz
-rw-r--r-- 1 dump dump   28K Sep 21 03:45 labswiki.redirect.sql.gz
-rw-r--r-- 1 dump dump   295 Sep 21 04:13 labswiki.revision_actor_temp-schema.sql.gz
-rw-r--r-- 1 dump dump  2.6M Sep 21 03:45 labswiki.revision_actor_temp.sql.gz
-rw-r--r-- 1 dump dump   252 Sep 21 04:13 labswiki.revision_comment_temp-schema.sql.gz
-rw-r--r-- 1 dump dump  1.6M Sep 21 03:45 labswiki.revision_comment_temp.sql.gz
-rw-r--r-- 1 dump dump   420 Sep 21 04:13 labswiki.revision-schema.sql.gz
-rw-r--r-- 1 dump dump   13M Sep 21 03:45 labswiki.revision.sql.gz
-rw-r--r-- 1 dump dump    89 Sep 21 03:20 labswiki-schema-create.sql.gz
-rw-r--r-- 1 dump dump   273 Sep 21 04:13 labswiki.searchindex-schema.sql.gz
-rw-r--r-- 1 dump dump  2.8M Sep 21 03:20 labswiki.searchindex.sql.gz
-rw-r--r-- 1 dump dump   254 Sep 21 04:13 labswiki.site_identifiers-schema.sql.gz
-rw-r--r-- 1 dump dump   387 Sep 21 04:13 labswiki.sites-schema.sql.gz
-rw-r--r-- 1 dump dump   270 Sep 21 04:13 labswiki.site_stats-schema.sql.gz
-rw-r--r-- 1 dump dump   166 Sep 21 03:45 labswiki.site_stats.sql.gz
-rw-r--r-- 1 dump dump   249 Sep 21 04:13 labswiki.slot_roles-schema.sql.gz
-rw-r--r-- 1 dump dump   149 Sep 21 03:45 labswiki.slot_roles.sql.gz
-rw-r--r-- 1 dump dump   267 Sep 21 04:13 labswiki.slots-schema.sql.gz
-rw-r--r-- 1 dump dump   14M Sep 21 03:45 labswiki.slots.sql.gz
-rw-r--r-- 1 dump dump   267 Sep 21 04:13 labswiki.spoofuser-schema.sql.gz
-rw-r--r-- 1 dump dump  280K Sep 21 03:45 labswiki.spoofuser.sql.gz
-rw-r--r-- 1 dump dump   292 Sep 21 04:13 labswiki.templatelinks-schema.sql.gz
-rw-r--r-- 1 dump dump  129K Sep 21 03:45 labswiki.templatelinks.sql.gz
-rw-r--r-- 1 dump dump   278 Sep 21 04:13 labswiki.text-schema.sql.gz
-rw-r--r-- 1 dump dump   17G Sep 21 04:19 labswiki.text.sql.gz
-rw-r--r-- 1 dump dump   354 Sep 21 04:13 labswiki.thread_history-schema.sql.gz
-rw-r--r-- 1 dump dump  3.7K Sep 21 03:45 labswiki.thread_history.sql.gz
-rw-r--r-- 1 dump dump   260 Sep 21 04:13 labswiki.thread_pending_relationship-schema.sql.gz
-rw-r--r-- 1 dump dump   283 Sep 21 04:13 labswiki.thread_reaction-schema.sql.gz
-rw-r--r-- 1 dump dump   519 Sep 21 04:13 labswiki.thread-schema.sql.gz
-rw-r--r-- 1 dump dump  1.2K Sep 21 03:45 labswiki.thread.sql.gz
-rw-r--r-- 1 dump dump   216 Sep 21 04:13 labswiki.updatelog-schema.sql.gz
-rw-r--r-- 1 dump dump   29K Sep 21 03:45 labswiki.updatelog.sql.gz
-rw-r--r-- 1 dump dump   497 Sep 21 04:13 labswiki.uploadstash-schema.sql.gz
-rw-r--r-- 1 dump dump  6.6K Sep 21 03:45 labswiki.uploadstash.sql.gz
-rw-r--r-- 1 dump dump   244 Sep 21 04:13 labswiki.user_former_groups-schema.sql.gz
-rw-r--r-- 1 dump dump   15K Sep 21 03:45 labswiki.user_former_groups.sql.gz
-rw-r--r-- 1 dump dump   266 Sep 21 04:13 labswiki.user_groups-schema.sql.gz
-rw-r--r-- 1 dump dump  1015 Sep 21 03:45 labswiki.user_groups.sql.gz
-rw-r--r-- 1 dump dump   280 Sep 21 04:13 labswiki.user_message_state-schema.sql.gz
-rw-r--r-- 1 dump dump   191 Sep 21 03:45 labswiki.user_message_state.sql.gz
-rw-r--r-- 1 dump dump   256 Sep 21 04:13 labswiki.user_newtalk-schema.sql.gz
-rw-r--r-- 1 dump dump   12K Sep 21 03:45 labswiki.user_newtalk.sql.gz
-rw-r--r-- 1 dump dump   254 Sep 21 04:13 labswiki.user_properties-schema.sql.gz
-rw-r--r-- 1 dump dump  802K Sep 21 03:45 labswiki.user_properties.sql.gz
-rw-r--r-- 1 dump dump   414 Sep 21 04:13 labswiki.user-schema.sql.gz
-rw-r--r-- 1 dump dump  2.4M Sep 21 03:45 labswiki.user.sql.gz
-rw-r--r-- 1 dump dump   238 Sep 21 04:13 labswiki.watchlist_expiry-schema.sql.gz
-rw-r--r-- 1 dump dump   189 Sep 21 03:45 labswiki.watchlist_expiry.sql.gz
-rw-r--r-- 1 dump dump   340 Sep 21 04:13 labswiki.watchlist-schema.sql.gz
-rw-r--r-- 1 dump dump  6.3M Sep 21 03:45 labswiki.watchlist.sql.gz
-rw-r--r-- 1 dump dump   386 Sep 21 04:13 labswiki.wikilove_log-schema.sql.gz
-rw-r--r-- 1 dump dump  6.3K Sep 21 03:45 labswiki.wikilove_log.sql.gz

I am waiting for the m5 backup now.

• Marostegui added a comment.Sep 21 2021, 5:38 AM

@jcrespo FYI

[07:36:48]  <+icinga-wm> PROBLEM - dump of s6 in codfw on alert1001 is CRITICAL: Last dump for s6 at codfw (db2141.codfw.wmnet:3316) taken on 2021-09-21 04:20:13 is 102 GB, but previous one was 85 GB, a change of 20.4% https://wikitech.wikimedia.org/wiki/MariaDB/Backups%23Alerting

[07:37:57]  <+icinga-wm> ACKNOWLEDGEMENT - dump of s6 in codfw on alert1001 is CRITICAL: Last dump for s6 at codfw (db2141.codfw.wmnet:3316) taken on 2021-09-21 04:20:13 is 102 GB, but previous one was 85 GB, a change of 20.4% Marostegui Wikitech migration https://wikitech.wikimedia.org/wiki/MariaDB/Backups%23Alerting
• Marostegui added a comment.Sep 21 2021, 6:23 AM

I have done a restore from s6.labswiki dumps to make sure everything is ok, and it is. I was able to select count(*) from all the restored tables.

• Marostegui added a comment.Sep 21 2021, 6:37 AM

m5 backup finished:

root@dbprov1003:/srv/backups/dumps/latest/dump.m5.2021-09-21--06-10-51#  ls -lh | grep labswiki
-rw-r--r-- 1 dump dump  271 Sep 21 06:11 labswiki.abuse_filter_action-schema.sql.gz
-rw-r--r-- 1 dump dump  243 Sep 21 06:10 labswiki.abuse_filter_action.sql.gz
-rw-r--r-- 1 dump dump  408 Sep 21 06:11 labswiki.abuse_filter_history-schema.sql.gz
-rw-r--r-- 1 dump dump 2.3K Sep 21 06:10 labswiki.abuse_filter_history.sql.gz
-rw-r--r-- 1 dump dump  483 Sep 21 06:11 labswiki.abuse_filter_log-schema.sql.gz
-rw-r--r-- 1 dump dump 7.3K Sep 21 06:10 labswiki.abuse_filter_log.sql.gz
-rw-r--r-- 1 dump dump  421 Sep 21 06:11 labswiki.abuse_filter-schema.sql.gz
-rw-r--r-- 1 dump dump 1.5K Sep 21 06:10 labswiki.abuse_filter.sql.gz
-rw-r--r-- 1 dump dump  279 Sep 21 06:11 labswiki.actor-schema.sql.gz
-rw-r--r-- 1 dump dump 331K Sep 21 06:10 labswiki.actor.sql.gz
-rw-r--r-- 1 dump dump  424 Sep 21 06:11 labswiki.archive-schema.sql.gz
-rw-r--r-- 1 dump dump  55M Sep 21 06:11 labswiki.archive.sql.gz
-rw-r--r-- 1 dump dump  250 Sep 21 06:11 labswiki.babel-schema.sql.gz
-rw-r--r-- 1 dump dump 2.2K Sep 21 06:10 labswiki.babel.sql.gz
-rw-r--r-- 1 dump dump  232 Sep 21 06:11 labswiki.betafeatures_user_counts-schema.sql.gz
-rw-r--r-- 1 dump dump  240 Sep 21 06:10 labswiki.betafeatures_user_counts.sql.gz
-rw-r--r-- 1 dump dump  223 Sep 21 06:11 labswiki.bv2017_edits-schema.sql.gz
-rw-r--r-- 1 dump dump 8.0K Sep 21 06:10 labswiki.bv2017_edits.sql.gz
-rw-r--r-- 1 dump dump  378 Sep 21 06:11 labswiki.categorylinks-schema.sql.gz
-rw-r--r-- 1 dump dump 227K Sep 21 06:10 labswiki.categorylinks.sql.gz
-rw-r--r-- 1 dump dump  295 Sep 21 06:11 labswiki.category-schema.sql.gz
-rw-r--r-- 1 dump dump 5.9K Sep 21 06:10 labswiki.category.sql.gz
-rw-r--r-- 1 dump dump  312 Sep 21 06:11 labswiki.change_tag_def-schema.sql.gz
-rw-r--r-- 1 dump dump  499 Sep 21 06:10 labswiki.change_tag_def.sql.gz
-rw-r--r-- 1 dump dump  323 Sep 21 06:11 labswiki.change_tag-schema.sql.gz
-rw-r--r-- 1 dump dump 136K Sep 21 06:10 labswiki.change_tag.sql.gz
-rw-r--r-- 1 dump dump  273 Sep 21 06:11 labswiki.comment-schema.sql.gz
-rw-r--r-- 1 dump dump 8.0M Sep 21 06:10 labswiki.comment.sql.gz
-rw-r--r-- 1 dump dump  232 Sep 21 06:11 labswiki.config-schema.sql.gz
-rw-r--r-- 1 dump dump  251 Sep 21 06:11 labswiki.content_models-schema.sql.gz
-rw-r--r-- 1 dump dump  240 Sep 21 06:10 labswiki.content_models.sql.gz
-rw-r--r-- 1 dump dump  285 Sep 21 06:11 labswiki.content-schema.sql.gz
-rw-r--r-- 1 dump dump  49M Sep 21 06:11 labswiki.content.sql.gz
-rw-r--r-- 1 dump dump  456 Sep 21 06:11 labswiki.cu_changes-schema.sql.gz
-rw-r--r-- 1 dump dump 1.1M Sep 21 06:10 labswiki.cu_changes.sql.gz
-rw-r--r-- 1 dump dump  378 Sep 21 06:11 labswiki.cu_log-schema.sql.gz
-rw-r--r-- 1 dump dump 4.4K Sep 21 06:10 labswiki.cu_log.sql.gz
-rw-r--r-- 1 dump dump  348 Sep 21 06:11 labswiki.discussiontools_subscription-schema.sql.gz
-rw-r--r-- 1 dump dump  327 Sep 21 06:11 labswiki.echo_email_batch-schema.sql.gz
-rw-r--r-- 1 dump dump  604 Sep 21 06:10 labswiki.echo_email_batch.sql.gz
-rw-r--r-- 1 dump dump  347 Sep 21 06:11 labswiki.echo_event-schema.sql.gz
-rw-r--r-- 1 dump dump 917K Sep 21 06:10 labswiki.echo_event.sql.gz
-rw-r--r-- 1 dump dump  369 Sep 21 06:11 labswiki.echo_notification-schema.sql.gz
-rw-r--r-- 1 dump dump 1.2M Sep 21 06:10 labswiki.echo_notification.sql.gz
-rw-r--r-- 1 dump dump  294 Sep 21 06:11 labswiki.echo_target_page-schema.sql.gz
-rw-r--r-- 1 dump dump 1.1M Sep 21 06:10 labswiki.echo_target_page.sql.gz
-rw-r--r-- 1 dump dump  326 Sep 21 06:11 labswiki.externallinks-schema.sql.gz
-rw-r--r-- 1 dump dump 961K Sep 21 06:10 labswiki.externallinks.sql.gz
-rw-r--r-- 1 dump dump  248 Sep 21 06:11 labswiki.external_user-schema.sql.gz
-rw-r--r-- 1 dump dump  639 Sep 21 06:11 labswiki.filearchive-schema.sql.gz
-rw-r--r-- 1 dump dump 9.3K Sep 21 06:10 labswiki.filearchive.sql.gz
-rw-r--r-- 1 dump dump  339 Sep 21 06:11 labswiki.geo_tags-schema.sql.gz
-rw-r--r-- 1 dump dump  370 Sep 21 06:11 labswiki.globalblocks-schema.sql.gz
-rw-r--r-- 1 dump dump  280 Sep 21 06:11 labswiki.global_block_whitelist-schema.sql.gz
-rw-r--r-- 1 dump dump  307 Sep 21 06:11 labswiki.globalimagelinks-schema.sql.gz
-rw-r--r-- 1 dump dump  272 Sep 21 06:11 labswiki.historical_thread-schema.sql.gz
-rw-r--r-- 1 dump dump  281 Sep 21 06:11 labswiki.imagelinks-schema.sql.gz
-rw-r--r-- 1 dump dump  47K Sep 21 06:10 labswiki.imagelinks.sql.gz
-rw-r--r-- 1 dump dump  547 Sep 21 06:11 labswiki.image-schema.sql.gz
-rw-r--r-- 1 dump dump 316K Sep 21 06:10 labswiki.image.sql.gz
-rw-r--r-- 1 dump dump  263 Sep 21 06:11 labswiki.interwiki-schema.sql.gz
-rw-r--r-- 1 dump dump  251 Sep 21 06:11 labswiki.ipblocks_restrictions-schema.sql.gz
-rw-r--r-- 1 dump dump  502 Sep 21 06:11 labswiki.ipblocks-schema.sql.gz
-rw-r--r-- 1 dump dump  77K Sep 21 06:10 labswiki.ipblocks.sql.gz
-rw-r--r-- 1 dump dump  273 Sep 21 06:11 labswiki.ip_changes-schema.sql.gz
-rw-r--r-- 1 dump dump  20K Sep 21 06:10 labswiki.ip_changes.sql.gz
-rw-r--r-- 1 dump dump  280 Sep 21 06:11 labswiki.iwlinks-schema.sql.gz
-rw-r--r-- 1 dump dump 208K Sep 21 06:10 labswiki.iwlinks.sql.gz
-rw-r--r-- 1 dump dump  402 Sep 21 06:11 labswiki.job-schema.sql.gz
-rw-r--r-- 1 dump dump 842K Sep 21 06:10 labswiki.job.sql.gz
-rw-r--r-- 1 dump dump  238 Sep 21 06:11 labswiki.l10n_cache-schema.sql.gz
-rw-r--r-- 1 dump dump  260 Sep 21 06:11 labswiki.langlinks-schema.sql.gz
-rw-r--r-- 1 dump dump 3.9K Sep 21 06:10 labswiki.langlinks.sql.gz
-rw-r--r-- 1 dump dump  257 Sep 21 06:11 labswiki.ldap_domains-schema.sql.gz
-rw-r--r-- 1 dump dump 271K Sep 21 06:10 labswiki.ldap_domains.sql.gz
-rw-r--r-- 1 dump dump  305 Sep 21 06:11 labswiki.linter-schema.sql.gz
-rw-r--r-- 1 dump dump  32K Sep 21 06:10 labswiki.linter.sql.gz
-rw-r--r-- 1 dump dump  441 Sep 21 06:11 labswiki.logging-schema.sql.gz
-rw-r--r-- 1 dump dump 6.5M Sep 21 06:11 labswiki.logging.sql.gz
-rw-r--r-- 1 dump dump  258 Sep 21 06:11 labswiki.log_search-schema.sql.gz
-rw-r--r-- 1 dump dump 122K Sep 21 06:10 labswiki.log_search.sql.gz
-rw-r--r-- 1 dump dump  271 Sep 21 06:11 labswiki.mathoid-schema.sql.gz
-rw-r--r-- 1 dump dump  233 Sep 21 06:11 labswiki.module_deps-schema.sql.gz
-rw-r--r-- 1 dump dump  81K Sep 21 06:10 labswiki.module_deps.sql.gz
-rw-r--r-- 1 dump dump  276 Sep 21 06:11 labswiki.oathauth_users_restore-schema.sql.gz
-rw-r--r-- 1 dump dump  17K Sep 21 06:10 labswiki.oathauth_users_restore.sql.gz
-rw-r--r-- 1 dump dump  231 Sep 21 06:11 labswiki.oathauth_users-schema.sql.gz
-rw-r--r-- 1 dump dump  82K Sep 21 06:10 labswiki.oathauth_users.sql.gz
-rw-r--r-- 1 dump dump  314 Sep 21 06:11 labswiki.oauth2_access_tokens-schema.sql.gz
-rw-r--r-- 1 dump dump  386 Sep 21 06:11 labswiki.oauth_accepted_consumer-schema.sql.gz
-rw-r--r-- 1 dump dump 1.7K Sep 21 06:10 labswiki.oauth_accepted_consumer.sql.gz
-rw-r--r-- 1 dump dump  562 Sep 21 06:11 labswiki.oauth_registered_consumer-schema.sql.gz
-rw-r--r-- 1 dump dump 2.0K Sep 21 06:10 labswiki.oauth_registered_consumer.sql.gz
-rw-r--r-- 1 dump dump  256 Sep 21 06:11 labswiki.objectcache-schema.sql.gz
-rw-r--r-- 1 dump dump  21K Sep 21 06:10 labswiki.objectcache.sql.gz
-rw-r--r-- 1 dump dump  553 Sep 21 06:11 labswiki.oldimage-schema.sql.gz
-rw-r--r-- 1 dump dump  29K Sep 21 06:10 labswiki.oldimage.sql.gz
-rw-r--r-- 1 dump dump  290 Sep 21 06:11 labswiki.pagelinks-schema.sql.gz
-rw-r--r-- 1 dump dump 262K Sep 21 06:10 labswiki.pagelinks.sql.gz
-rw-r--r-- 1 dump dump  278 Sep 21 06:11 labswiki.page_props-schema.sql.gz
-rw-r--r-- 1 dump dump  58K Sep 21 06:10 labswiki.page_props.sql.gz
-rw-r--r-- 1 dump dump  340 Sep 21 06:11 labswiki.page_restrictions-schema.sql.gz
-rw-r--r-- 1 dump dump 1.8K Sep 21 06:10 labswiki.page_restrictions.sql.gz
-rw-r--r-- 1 dump dump  433 Sep 21 06:11 labswiki.page-schema.sql.gz
-rw-r--r-- 1 dump dump 743K Sep 21 06:10 labswiki.page.sql.gz
-rw-r--r-- 1 dump dump  310 Sep 21 06:11 labswiki.protected_titles-schema.sql.gz
-rw-r--r-- 1 dump dump  291 Sep 21 06:10 labswiki.protected_titles.sql.gz
-rw-r--r-- 1 dump dump  243 Sep 21 06:11 labswiki.querycache_info-schema.sql.gz
-rw-r--r-- 1 dump dump  453 Sep 21 06:10 labswiki.querycache_info.sql.gz
-rw-r--r-- 1 dump dump  260 Sep 21 06:11 labswiki.querycache-schema.sql.gz
-rw-r--r-- 1 dump dump 446K Sep 21 06:10 labswiki.querycache.sql.gz
-rw-r--r-- 1 dump dump  298 Sep 21 06:11 labswiki.querycachetwo-schema.sql.gz
-rw-r--r-- 1 dump dump 1.6K Sep 21 06:10 labswiki.querycachetwo.sql.gz
-rw-r--r-- 1 dump dump  536 Sep 21 06:11 labswiki.recentchanges-schema.sql.gz
-rw-r--r-- 1 dump dump 115K Sep 21 06:10 labswiki.recentchanges.sql.gz
-rw-r--r-- 1 dump dump  293 Sep 21 06:11 labswiki.redirect-schema.sql.gz
-rw-r--r-- 1 dump dump  28K Sep 21 06:10 labswiki.redirect.sql.gz
-rw-r--r-- 1 dump dump  295 Sep 21 06:11 labswiki.revision_actor_temp-schema.sql.gz
-rw-r--r-- 1 dump dump 2.6M Sep 21 06:10 labswiki.revision_actor_temp.sql.gz
-rw-r--r-- 1 dump dump  252 Sep 21 06:11 labswiki.revision_comment_temp-schema.sql.gz
-rw-r--r-- 1 dump dump 1.6M Sep 21 06:10 labswiki.revision_comment_temp.sql.gz
-rw-r--r-- 1 dump dump  420 Sep 21 06:11 labswiki.revision-schema.sql.gz
-rw-r--r-- 1 dump dump  13M Sep 21 06:11 labswiki.revision.sql.gz
-rw-r--r-- 1 dump dump   89 Sep 21 06:10 labswiki-schema-create.sql.gz
-rw-r--r-- 1 dump dump  273 Sep 21 06:11 labswiki.searchindex-schema.sql.gz
-rw-r--r-- 1 dump dump 2.8M Sep 21 06:10 labswiki.searchindex.sql.gz
-rw-r--r-- 1 dump dump  254 Sep 21 06:11 labswiki.site_identifiers-schema.sql.gz
-rw-r--r-- 1 dump dump  387 Sep 21 06:11 labswiki.sites-schema.sql.gz
-rw-r--r-- 1 dump dump  270 Sep 21 06:11 labswiki.site_stats-schema.sql.gz
-rw-r--r-- 1 dump dump  168 Sep 21 06:10 labswiki.site_stats.sql.gz
-rw-r--r-- 1 dump dump  249 Sep 21 06:11 labswiki.slot_roles-schema.sql.gz
-rw-r--r-- 1 dump dump  149 Sep 21 06:10 labswiki.slot_roles.sql.gz
-rw-r--r-- 1 dump dump  267 Sep 21 06:11 labswiki.slots-schema.sql.gz
-rw-r--r-- 1 dump dump  14M Sep 21 06:11 labswiki.slots.sql.gz
-rw-r--r-- 1 dump dump  267 Sep 21 06:11 labswiki.spoofuser-schema.sql.gz
-rw-r--r-- 1 dump dump 279K Sep 21 06:10 labswiki.spoofuser.sql.gz
-rw-r--r-- 1 dump dump  292 Sep 21 06:11 labswiki.templatelinks-schema.sql.gz
-rw-r--r-- 1 dump dump 129K Sep 21 06:10 labswiki.templatelinks.sql.gz
-rw-r--r-- 1 dump dump  278 Sep 21 06:11 labswiki.text-schema.sql.gz
-rw-r--r-- 1 dump dump  17G Sep 21 06:31 labswiki.text.sql.gz
-rw-r--r-- 1 dump dump  354 Sep 21 06:11 labswiki.thread_history-schema.sql.gz
-rw-r--r-- 1 dump dump 3.7K Sep 21 06:10 labswiki.thread_history.sql.gz
-rw-r--r-- 1 dump dump  260 Sep 21 06:11 labswiki.thread_pending_relationship-schema.sql.gz
-rw-r--r-- 1 dump dump  283 Sep 21 06:11 labswiki.thread_reaction-schema.sql.gz
-rw-r--r-- 1 dump dump  519 Sep 21 06:11 labswiki.thread-schema.sql.gz
-rw-r--r-- 1 dump dump 1.2K Sep 21 06:10 labswiki.thread.sql.gz
-rw-r--r-- 1 dump dump  216 Sep 21 06:11 labswiki.updatelog-schema.sql.gz
-rw-r--r-- 1 dump dump  29K Sep 21 06:10 labswiki.updatelog.sql.gz
-rw-r--r-- 1 dump dump  496 Sep 21 06:11 labswiki.uploadstash-schema.sql.gz
-rw-r--r-- 1 dump dump 6.6K Sep 21 06:10 labswiki.uploadstash.sql.gz
-rw-r--r-- 1 dump dump  244 Sep 21 06:11 labswiki.user_former_groups-schema.sql.gz
-rw-r--r-- 1 dump dump  15K Sep 21 06:10 labswiki.user_former_groups.sql.gz
-rw-r--r-- 1 dump dump  266 Sep 21 06:11 labswiki.user_groups-schema.sql.gz
-rw-r--r-- 1 dump dump 1015 Sep 21 06:10 labswiki.user_groups.sql.gz
-rw-r--r-- 1 dump dump  280 Sep 21 06:11 labswiki.user_message_state-schema.sql.gz
-rw-r--r-- 1 dump dump  191 Sep 21 06:10 labswiki.user_message_state.sql.gz
-rw-r--r-- 1 dump dump  256 Sep 21 06:11 labswiki.user_newtalk-schema.sql.gz
-rw-r--r-- 1 dump dump  12K Sep 21 06:10 labswiki.user_newtalk.sql.gz
-rw-r--r-- 1 dump dump  254 Sep 21 06:11 labswiki.user_properties-schema.sql.gz
-rw-r--r-- 1 dump dump 799K Sep 21 06:10 labswiki.user_properties.sql.gz
-rw-r--r-- 1 dump dump  414 Sep 21 06:11 labswiki.user-schema.sql.gz
-rw-r--r-- 1 dump dump 2.4M Sep 21 06:10 labswiki.user.sql.gz
-rw-r--r-- 1 dump dump  238 Sep 21 06:11 labswiki.watchlist_expiry-schema.sql.gz
-rw-r--r-- 1 dump dump  189 Sep 21 06:10 labswiki.watchlist_expiry.sql.gz
-rw-r--r-- 1 dump dump  341 Sep 21 06:11 labswiki.watchlist-schema.sql.gz
-rw-r--r-- 1 dump dump 6.3M Sep 21 06:11 labswiki.watchlist.sql.gz
-rw-r--r-- 1 dump dump  386 Sep 21 06:11 labswiki.wikilove_log-schema.sql.gz
-rw-r--r-- 1 dump dump 6.3K Sep 21 06:10 labswiki.wikilove_log.sql.gz
root@dbprov1003:/srv/backups/dumps/latest/dump.m5.2021-09-21--06-10-51#
gerritbot added a comment.Sep 21 2021, 6:40 AM

Change 722544 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] valid_section.pp: Remove reference to s10

https://gerrit.wikimedia.org/r/722544

gerritbot added a comment.Sep 21 2021, 11:57 AM

Change 722544 merged by Marostegui:

[operations/puppet@production] valid_section.pp: Remove reference to s10

https://gerrit.wikimedia.org/r/722544

gerritbot added a comment.Sep 21 2021, 2:54 PM

Change 722273 merged by Marostegui:

[operations/puppet@production] sections.yaml: Remove s10 from codfw

https://gerrit.wikimedia.org/r/722273

Stashbot added a comment.Sep 21 2021, 3:09 PM

Mentioned in SAL (#wikimedia-operations) [2021-09-21T15:09:59Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Remove s10 from codfw T167973', diff saved to https://phabricator.wikimedia.org/P17307 and previous config saved to /var/cache/conftool/dbconfig/20210921-150958-marostegui.json

gerritbot added a comment.Sep 22 2021, 7:34 AM

Change 722817 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] dumps-*-m5.sql: Remove labswiki grants

https://gerrit.wikimedia.org/r/722817

gerritbot added a comment.Sep 22 2021, 8:06 AM

Change 722820 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] dbbackups: Remove dump grants from m5 for labswiki

https://gerrit.wikimedia.org/r/722820

gerritbot added a comment.Sep 22 2021, 8:07 AM

Change 722820 abandoned by Jcrespo:

[operations/puppet@production] dbbackups: Remove dump grants from m5 for labswiki

Reason:

duplicate of https://gerrit.wikimedia.org/r/c/operations/puppet/+/722817

https://gerrit.wikimedia.org/r/722820

gerritbot added a comment.Sep 22 2021, 8:13 AM

Change 722817 merged by Marostegui:

[operations/puppet@production] dumps-*-m5.sql: Remove labswiki grants

https://gerrit.wikimedia.org/r/722817

• Marostegui added a comment.Sep 22 2021, 8:25 AM

@Andrew @bd808 any objections to drop the (already renamed) tables from m5 next week?

Andrew added a comment.Sep 22 2021, 1:44 PM
In T167973#7371228, @Marostegui wrote:

@Andrew @bd808 any objections to drop the (already renamed) tables from m5 next week?

works for me!

• Marostegui added a comment.Sep 27 2021, 5:36 AM

Checked again, no errors about labswiki in the last week, so I am going to go ahead and start dropping tables.
(Going to take a last time backup from m5).

• Marostegui added a comment.Sep 27 2021, 5:55 AM

One last ad-hoc backup has been taken and left at:

db1117:/srv/tmp.m5/export-20210927-053719
Stashbot added a comment.Sep 27 2021, 5:56 AM

Mentioned in SAL (#wikimedia-operations) [2021-09-27T05:56:30Z] <marostegui> Drop labswiki from m5 T167973

• Marostegui added a comment.Sep 27 2021, 5:57 AM

labswiki has been dropped in m5:

root@db1128.eqiad.wmnet[labswiki]> show tables;
+--------------------------------------+
| Tables_in_labswiki                   |
+--------------------------------------+
| T167973_abuse_filter                 |
| T167973_abuse_filter_action          |
| T167973_abuse_filter_history         |
| T167973_abuse_filter_log             |
| T167973_actor                        |
| T167973_archive                      |
| T167973_babel                        |
| T167973_betafeatures_user_counts     |
| T167973_bv2017_edits                 |
| T167973_category                     |
| T167973_categorylinks                |
| T167973_change_tag                   |
| T167973_change_tag_def               |
| T167973_comment                      |
| T167973_config                       |
| T167973_content                      |
| T167973_content_models               |
| T167973_cu_changes                   |
| T167973_cu_log                       |
| T167973_discussiontools_subscription |
| T167973_echo_email_batch             |
| T167973_echo_event                   |
| T167973_echo_notification            |
| T167973_echo_target_page             |
| T167973_external_user                |
| T167973_externallinks                |
| T167973_filearchive                  |
| T167973_geo_tags                     |
| T167973_global_block_whitelist       |
| T167973_globalblocks                 |
| T167973_globalimagelinks             |
| T167973_historical_thread            |
| T167973_image                        |
| T167973_imagelinks                   |
| T167973_interwiki                    |
| T167973_ip_changes                   |
| T167973_ipblocks                     |
| T167973_ipblocks_restrictions        |
| T167973_iwlinks                      |
| T167973_job                          |
| T167973_l10n_cache                   |
| T167973_langlinks                    |
| T167973_ldap_domains                 |
| T167973_linter                       |
| T167973_log_search                   |
| T167973_logging                      |
| T167973_mathoid                      |
| T167973_module_deps                  |
| T167973_oathauth_users               |
| T167973_oathauth_users_restore       |
| T167973_oauth2_access_tokens         |
| T167973_oauth_accepted_consumer      |
| T167973_oauth_registered_consumer    |
| T167973_objectcache                  |
| T167973_oldimage                     |
| T167973_page                         |
| T167973_page_props                   |
| T167973_page_restrictions            |
| T167973_pagelinks                    |
| T167973_protected_titles             |
| T167973_querycache                   |
| T167973_querycache_info              |
| T167973_querycachetwo                |
| T167973_recentchanges                |
| T167973_redirect                     |
| T167973_revision                     |
| T167973_revision_actor_temp          |
| T167973_revision_comment_temp        |
| T167973_searchindex                  |
| T167973_site_identifiers             |
| T167973_site_stats                   |
| T167973_sites                        |
| T167973_slot_roles                   |
| T167973_slots                        |
| T167973_spoofuser                    |
| T167973_templatelinks                |
| T167973_text                         |
| T167973_thread                       |
| T167973_thread_history               |
| T167973_thread_pending_relationship  |
| T167973_thread_reaction              |
| T167973_updatelog                    |
| T167973_uploadstash                  |
| T167973_user                         |
| T167973_user_former_groups           |
| T167973_user_groups                  |
| T167973_user_message_state           |
| T167973_user_newtalk                 |
| T167973_user_properties              |
| T167973_watchlist                    |
| T167973_watchlist_expiry             |
| T167973_wikilove_log                 |
+--------------------------------------+
92 rows in set (0.001 sec)

root@db1128.eqiad.wmnet[labswiki]> drop database if exists labswiki;
Query OK, 92 rows affected (0.265 sec)

root@db1128.eqiad.wmnet[(none)]>
• Marostegui updated the task description. (Show Details)Sep 27 2021, 5:58 AM
• Marostegui added a comment.Sep 27 2021, 6:00 AM

Now it is time to clean up the users, I need to review all these grants:

root@db1128.eqiad.wmnet[mysql]> select user,host from user where user like 'wik%';
+-----------+----------------+
| User      | Host           |
+-----------+----------------+
| wikiadmin | %              |
| wikiuser  | %              |
| wikiadmin | 10.192.32.34   |
| wikiadmin | 10.192.32.7    |
| wikiadmin | 10.64.16.77    |
| wikiadmin | 10.64.32.28    |
| wikiadmin | 208.80.154.160 |
| wikiuser  | 208.80.154.160 |
| wikiadmin | 208.80.155.109 |
| wikiuser  | 208.80.155.109 |
| wikiadmin | localhost      |
| wikiuser  | localhost      |
+-----------+----------------+
12 rows in set (0.002 sec)
gerritbot added a comment.Sep 27 2021, 6:54 AM

Change 723986 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] production-m5.sql: Remove labswiki grants

https://gerrit.wikimedia.org/r/723986

gerritbot added a comment.Sep 27 2021, 7:58 AM

Change 723986 merged by Marostegui:

[operations/puppet@production] production-m5.sql: Remove labswiki grants

https://gerrit.wikimedia.org/r/723986

• Marostegui closed this task as Resolved.Sep 27 2021, 8:19 AM

All grants have been cleaned up.

It is with great pleasure that I am going to close this ancient task. Thanks very much to everyone who's helped here!! Much appreciated

Andrew added a comment.Sep 27 2021, 5:13 PM

Thank you @Marostegui !

bd808 mentioned this in T310795: Revive Labtestwikitech (formerly: Abolish labtestwikitech).Jun 21 2022, 1:09 AM
Maintenance_bot moved this task from In progress to Done on the DBA board.Jun 21 2022, 1:29 AM
jijiki mentioned this in T328768: Wikitech issues for datacentre switchover (March 2023).Feb 7 2023, 10:25 AM
gerritbot added a comment.Jul 2 2024, 12:36 PM

Change #708632 abandoned by RhinosF1:

[operations/software/conftool@master] remove s10 references

Reason:

https://gerrit.wikimedia.org/r/708632

Maintenance_bot removed a project: Patch-For-Review.Jul 2 2024, 1:32 PM
Reedy mentioned this in T374952: Figure why we can't dump labswiki, aka Wikitech.Tue, Sep 17, 1:18 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits